Hi Martin, This is a really good question, thanks for bringing it up. Normally I would not include "ddt" tests in the interop test suite. I have two reasons for that opinion: 1. All DDT tests will end up with the same idempotent_id (UUID for the test), which makes them hard to track. 2. The list of tests generated from that "ddt" data will change over time as we add new recordset types [1]. We will also need to be very careful that the dataset order does not change. That said, those are important tests for the functionality of the Designate service. Users will expect to be able to create records of those types if they are using a "OpenStack Powered Platform with DNS" cloud. So, yes, I think we should keep them. I would expect the list to expand over time, with [1] being a candidate for that in the near future. If the use of "ddt" is a big problem for the compliance testing, we can consider breaking these out into individual tests (likely to be duplicates to the existing "ddt" tests to maintain the legacy test UUIDs). Michael [1] https://review.opendev.org/c/openstack/designate/+/802190 On Wed, Sep 8, 2021 at 7:28 AM Martin Kopec <mkopec@redhat.com> wrote:
Hi designate team,
I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]: designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types
That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt.
We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3].
The question is, should interop track all the permutations of the test? See [4].
[1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2... [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af84... [3] https://review.opendev.org/c/osf/interop/+/806598 [4] https://review.opendev.org/c/osf/interop/+/807586
Thank you, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA