Open Stack

If the gate grouping is a subset of the tests from all projects, I'm not sure how much of an improvement there would be long term. Ideally, we'd always just run the tests we need to run, which isn't easy define. One approach that I've been using to keep run times low is to define integration test suites between projects (nova-cinder, nova-auth, etc) that we package separately from our standard functional tests. This lets my pipeline be (in the example of Nova): Nova-Smoke, and then all Nova tests, Nova-Cinder, Nova-Networks, and Nova-Auth integration jobs in parallel. It's still a manual way of deciding what to run, but it's relieved some of the pain of what tests to run from the pipeline.

Daryl
________________________________________
From: James E. Blair [jeblair at openstack.org]
Sent: Monday, May 13, 2013 1:52 PM
To: All Things QA.
Subject: Re: [openstack-qa] tempest run length - need a gate tag - call for     help

Sean Dague <sean at dague.net> writes:

> Any assistance would be good.
>
> Right now we really just need 'gate' attr added to basically all the
> non skipped methods, we can prune later. Once 'gate' looks to be ~
> full, we can flip over check and gate to use that.
>
> I think long term the approach we're going to need to go with is 3
> sets of tests:
>
> smoke (< 10 mins)
> gate (< 45 mins)
> full (everything)
>
> All projects gate on gate
>
> Periodic runs of full - daily, more often?
>
> Tempest check runs full (but not gate), it's advisory.
>
> Some on demand facility for people to run full.
>
> At this point I'm not adding my +2 to any more tests (only approving
> fixes to existing tests) until we get gate tag in, as I don't think we
> should be running any longer than we currently are.

We discussed this at the summit, and while running fewer tests is
certainly one of the things we can do, I don't remember consensus that
it was our first priority.

We have a number of other things that we can do to reduce run-time that
I think we agreed should be a higher priority:

A) Parallelize the test runner (move to testr).
B) Split the run into multiple jobs (XML vs JSON, etc).
C) Focus on flakey tests so that gate resets are less of a factor
   (reducing sensitivity to runtime).

Note that work on both A and B independently facilitates C.

I think the general direction we'd like to head is to run _more_ tests,
not less.  Further, I don't think that check jobs and gate jobs should
run different tests -- some people will learn to just ignore check jobs
and enqueue failing jobs into the gate (as people already ignore
non-voting jobs), resulting in more bad code landing.  It's also
optimizing the wrong pipeline -- developers are more sensitive to slow
check jobs than gate jobs.

I got the impression that we all agreed that testr was the highest
priority for this, and I'd still like to see that land before we move on
to functional job splits.  Is that effort progressing?  What can we do
to help?

-Jim

_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa