Open Stack

There’s been a bit of discussion about how to uniquely identify
tests since the idempotent_id patch landed in Tempest. I think
it’s important to clarify what the patch accomplished, and how
we can use it within the Defcore framework to identify tests.

The idempotent_ids give us a way to track a function that
provides a test. Within Tempest, each function is meant to
test one kind of behavior, but it’s possible for that behavior to
be run in different modes. For example, some networking tests
run in both an IPv4 and IPv6 mode. This is accomplished by
subclassing, which will create a new test name (distinct from
the function name and expanded path*) for that test. This means 
that there is not a 1:1 relationship between functions and test
names. It’s one of the reasons that the name ‘idempotent_id’
was used instead of ‘uuid’.

Now, at first glance this seems like it doesn’t help us in our goal
of tracking tests through refactoring. After a number of discussions
with members of the community about testing I don’t think that 
this is necessarily the case. A unique identifier already exists for
a given test: the test name. Although it may be pedantic I
would argue that changing a test name makes it a different test,
whether or not it actually changes in behavior.

What we have is two identifiers. A stable ‘idempotent_id’ that we can
use to track functionality, and possibly unstable test names that
uniquely identify the test we want to run. When this is combined with
the date of capability approval, which can be matched to a github
hash, this gives an exact representation of the tests that need to
be run for compatibility. It even gives us a means to parse the
Tempest tree and determine the latest and earliest points in time
that the Tempest tree can be reasonably applied to a Defcore
capabilities release.

Within our process we can now track three things. The functionality
which may be refactored to new locations but is identified by
idempotent_id, and the test name which may also change but
will have a stable idempotent_id associated with it, and the
Tempest git hash that tells us the point in time the capabilities
were defined in relation to the tests.

Technically I don’t think that trying to match unique identifiers to
tests is a worthwhile problem to try and solve. How do you
distinguish between a refactoring and a new test? How do you
guarantee that generated ids will actually be stable across
refactoring without being disturbed by possible new subclassing?
If feels like a big problem to solve for what we get back, and the
tools we now have in place give us much more flexibility
and power than we even had just a few weeks ago.

-Chris

* the unittest/testtools framework calls it the test_id, which the 
expanded path of the test method to which the attrs are
appended. It might be worthwhile to use this expanded name as
the defcore test name to help ease automation. Right now we think
of the test name as just the function name.