Open Stack

sounds good, unrolling the related tests from DDT is definitely one of the
solutions.

This DDT thing raises a few questions because uniquely identifying tests is
not just an interop requirement, it should be a general goal.
To highlight that I proposed a topic for the PTG in the designate's agenda
[1].

[1] https://etherpad.opendev.org/p/yoga-ptg-designate

On Fri, 10 Sept 2021 at 17:53, Arkady Kanevsky <akanevsk at redhat.com> wrote:

> I am good with that.
> But Martin should respond since it will impact Refstack.
>
> On Fri, Sep 10, 2021 at 10:46 AM Michael Johnson <johnsomor at gmail.com>
> wrote:
>
>> Personally I think it is cleaner to just unroll these from DDT into
>> individual stub tests. There are not a lot of them and they are
>> important for the interop testing, so should be cleanly tracked.
>> Frankly we have spent more time talking about the problems than the
>> change would take to write. grin
>>
>> If the interop team agrees, I am happy to do the work on the tests.
>>
>> Michael
>>
>> On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi
>> <gouthampravi at gmail.com> wrote:
>> >
>> >
>> >
>> > On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec at redhat.com> wrote:
>> >>
>> >> From interop perspective it's also better not to have multiple tests
>> with the same id.
>> >> We encountered one more problem with ddt - the test names seem not to
>> be generated consistently, see this:
>> >> https://paste.opendev.org/show/809187/
>> >> The test can have either _00009_TXT suffix or _9_TXT one.
>> >>
>> >> Until we figure this out, I think we will need to flag the test in
>> interop - so that a skip of the test (because of the name mismatch in this
>> case) won't make the whole guideline fail.
>> >>
>> >> Luigi's idea is great. Every test should be identified by a unique id
>> and it shouldn't matter that the test is generated (ddt). Different input
>> data -> different test -> different name -> different id.
>> >> Let's try to explore whether having a unique id per ddt entry is
>> possible.
>> >
>> >
>> >
>> > I think we could annotate the test inputs and include a unique UUID
>> included in the test name/doc per test input. Something along these lines:
>> https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
>> > However, adding the UUID as a test attr (a functionality of "testtools"
>> [1]) will require a hook within ddt likely.
>> >
>> > However, as we've discovered, there's no way to avoid adding an ordinal
>> into the generated test name when using DDT, and so we'll have to either
>> set a PYTHONHASHSEED and disable hash randomization, or ensure that values
>> appear in the same order all the time (example [2])
>> >
>> > [1]
>> https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892
>> > [2]
>> https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
>> >
>> >
>> >
>> >>
>> >>
>> >>
>> >> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano at redhat.com>
>> wrote:
>> >>>
>> >>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
>> >>> > If the use of "ddt" is a big problem for the compliance testing, we
>> >>> > can consider breaking these out into individual tests (likely to be
>> >>> > duplicates to the existing "ddt" tests to maintain the legacy test
>> >>> > UUIDs).
>> >>>
>> >>> I was wondering: wouldn't it be possible to expand or use ddt somehow
>> to
>> >>> inject different UUIDs for each generated test? You know in advance
>> how many
>> >>> tests are going to be generated.
>> >>>
>> >>> --
>> >>> Luigi
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Martin
>>
>>
>
> --
> Arkady Kanevsky, Ph.D.
> Phone: 972 707-6456
> Corporate Phone: 919 729-5744 ext. 8176456
>


-- 
Martin Kopec
Senior Software Quality Engineer
Red Hat EMEA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210913/85672a41/attachment-0001.htm>