Open Stack

Personally I think it is cleaner to just unroll these from DDT into
individual stub tests. There are not a lot of them and they are
important for the interop testing, so should be cleanly tracked.
Frankly we have spent more time talking about the problems than the
change would take to write. grin

If the interop team agrees, I am happy to do the work on the tests.

Michael

On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi
<gouthampravi at gmail.com> wrote:
>
>
>
> On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec at redhat.com> wrote:
>>
>> From interop perspective it's also better not to have multiple tests with the same id.
>> We encountered one more problem with ddt - the test names seem not to be generated consistently, see this:
>> https://paste.opendev.org/show/809187/
>> The test can have either _00009_TXT suffix or _9_TXT one.
>>
>> Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.
>>
>> Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id.
>> Let's try to explore whether having a unique id per ddt entry is possible.
>
>
>
> I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
> However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely.
>
> However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2])
>
> [1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892
> [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
>
>
>
>>
>>
>>
>> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano at redhat.com> wrote:
>>>
>>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
>>> > If the use of "ddt" is a big problem for the compliance testing, we
>>> > can consider breaking these out into individual tests (likely to be
>>> > duplicates to the existing "ddt" tests to maintain the legacy test
>>> > UUIDs).
>>>
>>> I was wondering: wouldn't it be possible to expand or use ddt somehow to
>>> inject different UUIDs for each generated test? You know in advance how many
>>> tests are going to be generated.
>>>
>>> --
>>> Luigi
>>>
>>>
>>
>>
>> --
>> Martin