Open Stack

On 01/04/2013 09:34 AM, David Kranz wrote:
> It's been several months since we agreed to turn on the full gate and I
> wanted to review where we are and suggest some next steps.
>
> There were two obstacles: some flaky test failures and performance
> issues. Thanks to the good work of some IBM folks, the
> flaky problem has been squashed. The tempest hourly run has been doing
> well, just failing sometimes in ways that have nothing
> to do with tempest.
>
> On the performance side there are ongoing efforts to get rid of nose and
> support parallel test execution, but the end result of that is
> not yet visible and I think will not come online in the near future.
> Meanwhile, the number of tests has continued
> to increase. This is good, but the current tests take more than an hour
> to run and the time keeps going up. Also, the technical committee
> seems to be taking a more expansive view of projects that will need
> inclusion in tempest such as heat, ceilometer, etc.

I still think we'll have this ready for Grizzly (by H Summit), but I'm 
with you in feeling the pain now.

> In the long run, or perhaps even in the medium run, if tempest succeeds
> in its mission to provide complete integration
> and api stability testing for all openstack proejcts,  it does not make
> sense to run every tempest test
> on every commit in every project. I believe this it true even if we have
> parallel test execution. Here is a proposal that
> attempts to create a scalable methodology:
>
> 1. Categorize all tests as one of:
>   a) functional (default)
>   b) integration (selection of tests that interact with other projects)
>   c) slow (too slow to run on every commit, might be using multiple
> nodes, etc.)

+1

> 2. Make sure that each project that is supported by openstack/tempest
> has all of its tests in one directory. This is
>      almost true already.

Yeh, I think we're doing pretty good on that one.

> 3. On a commit to project X, run:
>     a) all functional tests in the project X directory, excepting those
> that are 'slow'
>     b) all integration tests

A bunch of the "slow" is actually probably possible to re-factor in the 
current tree to be smarter. Part of removing the docstrings was to make 
it a little easier to look at the expensive classes.

> 4. Create an hourly run for each project that runs all its tests to
> check the slow ones and as
> a matter of goodness. Since periodic tests are useless if no one looks
> at the failures, designate a rotation of
> people to examine failures and act on them. For this to work, the
> project core developers need to commit to
> treating bugs that slip past the gate with highest priority.

As much as I'd like this to work, I just don't think that it will. Tests 
that prevent merges get dealt with, ones that don't just don't get priority.

> 5. Continue to run full tempest on every tempest commit until we can't
> bear it any longer.

Yes, fair.

      -Sean

-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com