Open Stack

+1 but I have some restrictions regarding #4

I think we should kick of the hourly project build only If there is  any commit for that track within last hour.  We need a script to check the commit history to kick-off the hourly Jenkins. We implemented something similar at LinkedIn and it saved us a lot of build-debug time.

We need some automation to integrate the test failures with automatic ticketing to facilitate easy debugging.  The proposal is, if we use test assertions with right flags/priority levels (similar to log-4j levels), then based on failed test results we can automatically prioritize them in different buckets - blocker, critical, major and minor and fix the issues based on priority. I wonder if testtools does provide us with such assertion levels, need to check.

Just a note, I am a recent addition to Cloud OS. I assumed this thread is meant for open discussion.

Thanks,
Bageshree Lokare
HP Cloud


-----Original Message-----
From: David Kranz [mailto:david.kranz at qrclab.com] 
Sent: Friday, January 04, 2013 6:34 AM
To: openstack-qa at lists.openstack.org
Subject: [openstack-qa] Turning on the full tempest gate

It's been several months since we agreed to turn on the full gate and I wanted to review where we are and suggest some next steps.

There were two obstacles: some flaky test failures and performance issues. Thanks to the good work of some IBM folks, the flaky problem has been squashed. The tempest hourly run has been doing well, just failing sometimes in ways that have nothing to do with tempest.

On the performance side there are ongoing efforts to get rid of nose and support parallel test execution, but the end result of that is not yet visible and I think will not come online in the near future. 
Meanwhile, the number of tests has continued to increase. This is good, but the current tests take more than an hour to run and the time keeps going up. Also, the technical committee seems to be taking a more expansive view of projects that will need inclusion in tempest such as heat, ceilometer, etc.

In the long run, or perhaps even in the medium run, if tempest succeeds in its mission to provide complete integration and api stability testing for all openstack proejcts,  it does not make sense to run every tempest test on every commit in every project. I believe this it true even if we have parallel test execution. Here is a proposal that attempts to create a scalable methodology:

1. Categorize all tests as one of:
  a) functional (default)
  b) integration (selection of tests that interact with other projects)
  c) slow (too slow to run on every commit, might be using multiple nodes, etc.)

2. Make sure that each project that is supported by openstack/tempest has all of its tests in one directory. This is
     almost true already.

3. On a commit to project X, run:
    a) all functional tests in the project X directory, excepting those that are 'slow'
    b) all integration tests

4. Create an hourly run for each project that runs all its tests to check the slow ones and as a matter of goodness. Since periodic tests are useless if no one looks at the failures, designate a rotation of people to examine failures and act on them. For this to work, the project core developers need to commit to treating bugs that slip past the gate with highest priority.


5. Continue to run full tempest on every tempest commit until we can't bear it any longer.

Comments welcome.

  -David

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa