Open Stack

Hi Martin,

This is a really good question, thanks for bringing it up.

Normally I would not include "ddt" tests in the interop test suite. I
have two reasons for that opinion:
1. All DDT tests will end up with the same idempotent_id (UUID for the
test), which makes them hard to track.
2. The list of tests generated from that "ddt" data will change over
time as we add new recordset types [1]. We will also need to be very
careful that the dataset order does not change.

That said, those are important tests for the functionality of the
Designate service. Users will expect to be able to create records of
those types if they are using a "OpenStack Powered Platform with DNS"
cloud.

So, yes, I think we should keep them. I would expect the list to
expand over time, with [1] being a candidate for that in the near
future.
If the use of "ddt" is a big problem for the compliance testing, we
can consider breaking these out into individual tests (likely to be
duplicates to the existing "ddt" tests to maintain the legacy test
UUIDs).

Michael

[1] https://review.opendev.org/c/openstack/designate/+/802190

On Wed, Sep 8, 2021 at 7:28 AM Martin Kopec <mkopec at redhat.com> wrote:
>
> Hi designate team,
>
> I'm reaching out because we have encountered a problem with one of the tests [1]. We track the following name [2]:
> designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest.test_create_all_recordset_types
>
> That test name is actually incorrect due to the ddt and the real name of the test is created dynamically based on the test data passed via ddt.
>
> We have a consistency check which is supposed to verify that the test names tracked in our guidelines match the ones in Tempest/tempest plugins, however, there was a bug - we are fixing that by [3].
>
> The question is, should interop track all the permutations of the test? See [4].
>
> [1] https://opendev.org/openstack/designate-tempest-plugin/src/commit/da27a70ae2b39695ef6f03bbefb55afeacf1cdf3/designate_tempest_plugin/tests/api/v2/test_recordset.py#L87
> [2] https://opendev.org/osf/interop/src/commit/d8eac71fa507c68e9c1f83a19dd77af8483d01b6/add-ons/guidelines/dns.2020.11.json#L116-L117
> [3] https://review.opendev.org/c/osf/interop/+/806598
> [4] https://review.opendev.org/c/osf/interop/+/807586
>
> Thank you,
> --
> Martin Kopec
> Senior Software Quality Engineer
> Red Hat EMEA
>
>
>