Open Stack

+1

-----Original Message-----
From: David Kranz [mailto:david.kranz at qrclab.com] 
Sent: 04 January 2013 14:34
To: openstack-qa at lists.openstack.org
Subject: [openstack-qa] Turning on the full tempest gate

It's been several months since we agreed to turn on the full gate and I
wanted to review where we are and suggest some next steps.

There were two obstacles: some flaky test failures and performance issues.
Thanks to the good work of some IBM folks, the flaky problem has been
squashed. The tempest hourly run has been doing well, just failing sometimes
in ways that have nothing to do with tempest.

On the performance side there are ongoing efforts to get rid of nose and
support parallel test execution, but the end result of that is not yet
visible and I think will not come online in the near future. 
Meanwhile, the number of tests has continued to increase. This is good, but
the current tests take more than an hour to run and the time keeps going up.
Also, the technical committee seems to be taking a more expansive view of
projects that will need inclusion in tempest such as heat, ceilometer, etc.

In the long run, or perhaps even in the medium run, if tempest succeeds in
its mission to provide complete integration and api stability testing for
all openstack proejcts,  it does not make sense to run every tempest test on
every commit in every project. I believe this it true even if we have
parallel test execution. Here is a proposal that attempts to create a
scalable methodology:

1. Categorize all tests as one of:
  a) functional (default)
  b) integration (selection of tests that interact with other projects)
  c) slow (too slow to run on every commit, might be using multiple nodes,
etc.)

2. Make sure that each project that is supported by openstack/tempest has
all of its tests in one directory. This is
     almost true already.

3. On a commit to project X, run:
    a) all functional tests in the project X directory, excepting those that
are 'slow'
    b) all integration tests

4. Create an hourly run for each project that runs all its tests to check
the slow ones and as a matter of goodness. Since periodic tests are useless
if no one looks at the failures, designate a rotation of people to examine
failures and act on them. For this to work, the project core developers need
to commit to treating bugs that slip past the gate with highest priority.


5. Continue to run full tempest on every tempest commit until we can't bear
it any longer.

Comments welcome.

  -David

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6202 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20130104/255b706b/attachment.bin>