On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec@redhat.com> wrote:
From interop perspective it's also better not to have multiple tests with the same id.
We encountered one more problem with ddt - the test names seem not to be generated consistently, see this:
The test can have either _00009_TXT suffix or _9_TXT one.

Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.

Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id.
Let's try to explore whether having a unique id per ddt entry is possible.


I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely.  

However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2])

[1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc9d68320330aa/testtools/testcase.py#L892
[2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859


 


On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com> wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
> If the use of "ddt" is a big problem for the compliance testing, we
> can consider breaking these out into individual tests (likely to be
> duplicates to the existing "ddt" tests to maintain the legacy test
> UUIDs).

I was wondering: wouldn't it be possible to expand or use ddt somehow to
inject different UUIDs for each generated test? You know in advance how many
tests are going to be generated.

--
Luigi




--
Martin