Open Stack

Just had a long conversation with David Kranz about the "right" way to 
test API stability (among other things).  Just as accounting has double 
entry as a best practice to keep the bookkeepers honest, keeping the 
acceptance tests in a separate repository from the main project keeps us 
developers honest.

In Keystone, we have a growing body of tests that are for checking API 
stability.  A great example is the recent commit by Dolph for tests that 
confirm the behavior of the Trusts patch.  We made the decision as a 
team to not commit the trusts patch until the tests all ran, and had 
both up for review on gerrit at the same time. This was a good "honor 
system" approach.

David made the point earlier that Tempest has a lot of overlap with 
functional tests in the various projects.  We'd like to reduce the 
overlap.  As such, I think it is safe to say that if a test can live in 
either Tempest or its home project, the preference should be given to 
Tempest.  The caveat being, of course, that since tempest is run on each 
post to Gerrit, it should be treated as a scarce resource, and thus only 
tests that are have a wide applicability should be run on every commit 
across Open Stack.

Based on that, here is what I propose doing for Keystone.


1.  Create a set of incubation tests in Tempest that are not run by 
default.  When a new feature is proposed for Keystone, the developer 
will first submit an incubation test to Tempest.  This test should be 
signed off by a core Keystone developer (other than the author) as well 
as a tempest developer.

2.  Modify keystone/run_tests.sh  so that it can run the keystone subset 
of tests in tempest.  This will take an additional switch at first 
--tempest.  If this switch is passed, it will look for ../tempest and, 
if it does not exist, will inform the developer that they need to check 
out tempest.  If it does exist, it will populate the tempest config file 
with enough information to find a Keystone server set up IAW how 
keystone/tests/test_content_types.py sets up the server.

3. Extend keystone/run_tests.sh code will check for a scoreboard of 
incubated tests to add to the run in tempest. Part of a commit in the 
future will be adding the appropriate feature tests to this file when 
submitting to Gerrit.

4.  In the future, make running the tempest tests will be the default 
behavior, with an override switch --unit that will run only the tests 
under keystone/tests.

5.  after tempest is default, start migrating tests from test/keystone 
to tempest.

6.  Establish the workflow to promote incubated tests to gating tests.

All the above is predicated on being able to run a subset of the tempest 
tests that are appropriate for Keystone.  I am not suggesting that the 
entire body of Tempest tests should or even can be run by a Keystone 
developer.