Open Stack

It's been several months since we agreed to turn on the full gate and I 
wanted to review where we are and suggest some next steps.

There were two obstacles: some flaky test failures and performance 
issues. Thanks to the good work of some IBM folks, the
flaky problem has been squashed. The tempest hourly run has been doing 
well, just failing sometimes in ways that have nothing
to do with tempest.

On the performance side there are ongoing efforts to get rid of nose and 
support parallel test execution, but the end result of that is
not yet visible and I think will not come online in the near future. 
Meanwhile, the number of tests has continued
to increase. This is good, but the current tests take more than an hour 
to run and the time keeps going up. Also, the technical committee
seems to be taking a more expansive view of projects that will need 
inclusion in tempest such as heat, ceilometer, etc.

In the long run, or perhaps even in the medium run, if tempest succeeds 
in its mission to provide complete integration
and api stability testing for all openstack proejcts,  it does not make 
sense to run every tempest test
on every commit in every project. I believe this it true even if we have 
parallel test execution. Here is a proposal that
attempts to create a scalable methodology:

1. Categorize all tests as one of:
  a) functional (default)
  b) integration (selection of tests that interact with other projects)
  c) slow (too slow to run on every commit, might be using multiple 
nodes, etc.)

2. Make sure that each project that is supported by openstack/tempest 
has all of its tests in one directory. This is
     almost true already.

3. On a commit to project X, run:
    a) all functional tests in the project X directory, excepting those 
that are 'slow'
    b) all integration tests

4. Create an hourly run for each project that runs all its tests to 
check the slow ones and as
a matter of goodness. Since periodic tests are useless if no one looks 
at the failures, designate a rotation of
people to examine failures and act on them. For this to work, the 
project core developers need to commit to
treating bugs that slip past the gate with highest priority.


5. Continue to run full tempest on every tempest commit until we can't 
bear it any longer.

Comments welcome.

  -David