[designate][interop] test_create_all_recordset_types ddt issue
Hi designate team, I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]: designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt. We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3]. The question is, should interop track all the permutations of the test? See [4]. [1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2... [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af84... [3] https://review.opendev.org/c/osf/interop/+/806598 [4] https://review.opendev.org/c/osf/interop/+/807586 Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
Hi Martin, This is a really good question, thanks for bringing it up. Normally I would not include "ddt" tests in the interop test suite. I have two reasons for that opinion: 1. All DDT tests will end up with the same idempotent_id (UUID for the test), which makes them hard to track. 2. The list of tests generated from that "ddt" data will change over time as we add new recordset types [1]. We will also need to be very careful that the dataset order does not change. That said, those are important tests for the functionality of the Designate service. Users will expect to be able to create records of those types if they are using a "OpenStack Powered Platform with DNS" cloud. So, yes, I think we should keep them. I would expect the list to expand over time, with [1] being a candidate for that in the near future. If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs). Michael [1] https://review.opendev.org/c/openstack/designate/+/802190 On Wed, Sep 8, 2021 at 7:28 AM Martin Kopec <mkopec@redhat.com> wrote:
Hi designate team,
I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]: designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types
That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt.
We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3].
The question is, should interop track all the permutations of the test? See [4].
[1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2... [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af84... [3] https://review.opendev.org/c/osf/interop/+/806598 [4] https://review.opendev.org/c/osf/interop/+/807586
Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow to inject different UUIDs for each generated test? You know in advance how many tests are going to be generated. -- Luigi
From interop perspective it's also better not to have multiple tests with the same id. We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one.
Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail. Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is possible. On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com> wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow to inject different UUIDs for each generated test? You know in advance how many tests are going to be generated.
-- Luigi
-- Martin
On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec@redhat.com> wrote:
From interop perspective it's also better not to have multiple tests with the same id. We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one.
Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.
Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is possible.
I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely. However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2]) [1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc... [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com> wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow to inject different UUIDs for each generated test? You know in advance how many tests are going to be generated.
-- Luigi
-- Martin
Personally I think it is cleaner to just unroll these from DDT into individual stub tests. There are not a lot of them and they are important for the interop testing, so should be cleanly tracked. Frankly we have spent more time talking about the problems than the change would take to write. grin If the interop team agrees, I am happy to do the work on the tests. Michael On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec@redhat.com> wrote:
From interop perspective it's also better not to have multiple tests with the same id. We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one.
Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.
Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is possible.
I think we could annotate the test inputs and include a unique UUID included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114 However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely.
However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2])
[1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc... [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com> wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow to inject different UUIDs for each generated test? You know in advance how many tests are going to be generated.
-- Luigi
-- Martin
I am good with that. But Martin should respond since it will impact Refstack. On Fri, Sep 10, 2021 at 10:46 AM Michael Johnson <johnsomor@gmail.com> wrote:
Personally I think it is cleaner to just unroll these from DDT into individual stub tests. There are not a lot of them and they are important for the interop testing, so should be cleanly tracked. Frankly we have spent more time talking about the problems than the change would take to write. grin
If the interop team agrees, I am happy to do the work on the tests.
Michael
On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec@redhat.com> wrote:
From interop perspective it's also better not to have multiple tests
We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one.
Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.
Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is
with the same id. possible.
I think we could annotate the test inputs and include a unique UUID
included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely.
However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2])
[1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc... [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com>
wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow
to
inject different UUIDs for each generated test? You know in advance how many tests are going to be generated.
-- Luigi
-- Martin
-- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456
sounds good, unrolling the related tests from DDT is definitely one of the solutions. This DDT thing raises a few questions because uniquely identifying tests is not just an interop requirement, it should be a general goal. To highlight that I proposed a topic for the PTG in the designate's agenda [1]. [1] https://etherpad.opendev.org/p/yoga-ptg-designate On Fri, 10 Sept 2021 at 17:53, Arkady Kanevsky <akanevsk@redhat.com> wrote:
I am good with that. But Martin should respond since it will impact Refstack.
On Fri, Sep 10, 2021 at 10:46 AM Michael Johnson <johnsomor@gmail.com> wrote:
Personally I think it is cleaner to just unroll these from DDT into individual stub tests. There are not a lot of them and they are important for the interop testing, so should be cleanly tracked. Frankly we have spent more time talking about the problems than the change would take to write. grin
If the interop team agrees, I am happy to do the work on the tests.
Michael
On Thu, Sep 9, 2021 at 10:07 AM Goutham Pacha Ravi <gouthampravi@gmail.com> wrote:
On Thu, Sep 9, 2021 at 12:57 AM Martin Kopec <mkopec@redhat.com> wrote:
From interop perspective it's also better not to have multiple tests
We encountered one more problem with ddt - the test names seem not to be generated consistently, see this: https://paste.opendev.org/show/809187/ The test can have either _00009_TXT suffix or _9_TXT one.
Until we figure this out, I think we will need to flag the test in interop - so that a skip of the test (because of the name mismatch in this case) won't make the whole guideline fail.
Luigi's idea is great. Every test should be identified by a unique id and it shouldn't matter that the test is generated (ddt). Different input data -> different test -> different name -> different id. Let's try to explore whether having a unique id per ddt entry is
with the same id. possible.
I think we could annotate the test inputs and include a unique UUID
included in the test name/doc per test input. Something along these lines: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/808114
However, adding the UUID as a test attr (a functionality of "testtools" [1]) will require a hook within ddt likely.
However, as we've discovered, there's no way to avoid adding an ordinal into the generated test name when using DDT, and so we'll have to either set a PYTHONHASHSEED and disable hash randomization, or ensure that values appear in the same order all the time (example [2])
[1] https://github.com/testing-cabal/testtools/blob/f38af6765d4e412b7a7ed1c77ddc... [2] https://review.opendev.org/c/openstack/manila-tempest-plugin/+/755859
On Wed, 8 Sept 2021 at 18:15, Luigi Toscano <ltoscano@redhat.com>
wrote:
On Wednesday, 8 September 2021 17:48:30 CEST Michael Johnson wrote:
If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs).
I was wondering: wouldn't it be possible to expand or use ddt somehow
to
inject different UUIDs for each generated test? You know in advance how many tests are going to be generated.
-- Luigi
-- Martin
-- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456
-- Martin Kopec Senior Software Quality Engineer Red Hat EMEA
participants (5)
-
Arkady Kanevsky
-
Goutham Pacha Ravi
-
Luigi Toscano
-
Martin Kopec
-
Michael Johnson