Open Stack

On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec at redhat.com> wrote:

> From interop perspective it's also better not to have multiple tests with
> the same id.
> We encountered one more problem with ddt - the test names seem not to be
> generated consistently, see this:
> https://paste.opendev.org/show/809187/
> The test can have either _00009_TXT suffix or _9_TXT one.
>
> Until we figure this out, I think we will need to flag the test in interop
> - so that a skip of the test (because of the name mismatch in this case)
> won't make the whole guideline fail.
>
> Luigi's idea is great. Every test should be identified by a unique id and
> it shouldn't matter that the test is generated (ddt). Different input data
> -> different test -> different name -> different id.
> Let's try to explore whether having a unique id per ddt entry is possible.
>


I think we could annotate the test inputs and include a unique UUID
included in the test name/doc per test input. Something along these lines:
https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
However, adding the UUID as a test attr (a functionality of "testtools"
[1]) will require a hook within ddt likely.

However, as we've discovered, there's no way to avoid adding an ordinal
into the generated test name when using DDT, and so we'll have to either
set a PYTHONHASHSEED and disable hash randomization, or ensure that values
appear in the same order all the time (example [2])

[1]
https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892
[2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859




>
>
> On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano at redhat.com> wrote:
>
>> On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
>> > If the use of "ddt" is a big problem for the compliance testing, we
>> > can consider breaking these out into individual tests (likely to be
>> > duplicates to the existing "ddt" tests to maintain the legacy test
>> > UUIDs).
>>
>> I was wondering: wouldn't it be possible to expand or use ddt somehow to
>> inject different UUIDs for each generated test? You know in advance how
>> many
>> tests are going to be generated.
>>
>> --
>> Luigi
>>
>>
>>
>
> --
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210909/30fc241f/attachment-0001.html>