Open Stack

Nothing, just someone needs to sign up to do the work. The restructure 
will help a little with that. 
https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/tempest-repo-restructure,n,z

At some point it gets inefficient for us to manually slice the buckets, 
because then someone needs to manually retune them from time to time.

But I would not be opposed to such patches / proposals.

	-Sean

On 05/14/2013 05:13 PM, Daryl Walleck wrote:
> Just a thought, but even if Nose's parallelization is broken, what's stopping us from using Nose's loader and runner programmatically to kick individual test modules off to run in their own thread process (as a stop gap solution)? We're doing something similar (using just base unittest) with CloudCafe and it required no modifications to our tests. I couldn't see it needing modifications in this case either.
>
> Daryl
> ________________________________________
> From: David Kranz [david.kranz at qrclab.com]
> Sent: Tuesday, May 14, 2013 12:51 PM
> To: All Things QA.
> Cc: Sean Dague; Robert Collins
> Subject: Re: [openstack-qa] tempest run length - need a gate tag - call for help
>
> This subject seems to be as controversial on the list as it was at the
> summit. But I think we did agree (or agree to disagree) at the summit on
> the following points:
>
> 1. We should test as much as we can and not assume any "hardware"
> limitations unless they are proven to exist.
> 2. The tests need to run below some budget. It is not clear what it
> should be but many felt that the current time was near the upper bound.
> Others disagree.
> 3. We need parallel test execution ASAP
>
> The trouble was that (3) is not so easy or it would have been done
> months ago as this is a long-running problem. If Robert can help that
> would be great. If we solve this,
> the other issues will largely disappear.
>
>    -David
>
> On 5/14/2013 12:18 PM, Monty Taylor wrote:
>>
>> On 05/13/2013 12:51 PM, Sean Dague wrote:
>>> On 05/13/2013 02:52 PM, James E. Blair wrote:
>>>> Sean Dague <sean at dague.net> writes:
>>>>
>>>>> Any assistance would be good.
>>>>>
>>>>> Right now we really just need 'gate' attr added to basically all the
>>>>> non skipped methods, we can prune later. Once 'gate' looks to be ~
>>>>> full, we can flip over check and gate to use that.
>>>>>
>>>>> I think long term the approach we're going to need to go with is 3
>>>>> sets of tests:
>>>>>
>>>>> smoke (< 10 mins)
>>>>> gate (< 45 mins)
>>>>> full (everything)
>>>>>
>>>>> All projects gate on gate
>>>>>
>>>>> Periodic runs of full - daily, more often?
>>>>>
>>>>> Tempest check runs full (but not gate), it's advisory.
>>>>>
>>>>> Some on demand facility for people to run full.
>>>>>
>>>>> At this point I'm not adding my +2 to any more tests (only approving
>>>>> fixes to existing tests) until we get gate tag in, as I don't think we
>>>>> should be running any longer than we currently are.
>>>> We discussed this at the summit, and while running fewer tests is
>>>> certainly one of the things we can do, I don't remember consensus that
>>>> it was our first priority.
>>> It's not really about running fewer tests, it's about test growth.
>>> Tempest isn't just a tool for the gate, lots of people want to use it on
>>> their clouds. So at some point we do need to divorce the gate set from
>>> the whole set. Otherwise the tolerable length of the OpenStack gate
>>> limits the content people can contribute to test their real world clouds.
>>>
>>> We already have instances of this, stress tests, which are in tempest
>>> but not in the gate. David and I had some lunch time conversations about
>>> this one day and I think that we were both of the mindset that:
>>>    * not everything in tempest has to be in the gate
>>>    * however, everything in tempest has to get run on some interval for
>>> bitrot reasons (which we aren't currently doing)
>> I think we're still suffering from diverging assumptions about some
>> things. IMO, there are three types of tests:
>>
>> functional tests - test that a thing works
>> stress tests - hammer the thing to make sure it can handle load
>> performance regression tests - check that this commit performs as well
>> as the last one
>>
>> We are fundamentally unequipped to run the second two in any meaningful
>> way as part of OpenStack Infra right now - and I think that's ok for
>> now. But let's ignore them for the rest of this - they're a topic for
>> future.
>>
>> Our discussions about tests in the gate should at the moment be around
>> tests that test functionality and what to do with them.
>>
>> I am of the opinion that we should run every single functionality test
>> we have at our disposal every time we run anything, because that's how
>> we ensure that OpenStack works, and that is the current purpose of and
>> scope of the gate.
>>
>>>> We have a number of other things that we can do to reduce run-time that
>>>> I think we agreed should be a higher priority:
>>>>
>>>> A) Parallelize the test runner (move to testr).
>>>> B) Split the run into multiple jobs (XML vs JSON, etc).
>>>> C) Focus on flakey tests so that gate resets are less of a factor
>>>>       (reducing sensitivity to runtime).
>>>>
>>>> Note that work on both A and B independently facilitates C.
>>>>
>>>> I think the general direction we'd like to head is to run _more_ tests,
>>>> not less.  Further, I don't think that check jobs and gate jobs should
>>>> run different tests -- some people will learn to just ignore check jobs
>>>> and enqueue failing jobs into the gate (as people already ignore
>>>> non-voting jobs), resulting in more bad code landing.  It's also
>>>> optimizing the wrong pipeline -- developers are more sensitive to slow
>>>> check jobs than gate jobs.
>>>>
>>>> I got the impression that we all agreed that testr was the highest
>>>> priority for this, and I'd still like to see that land before we move on
>>>> to functional job splits.  Is that effort progressing?  What can we do
>>>> to help?
>>> A is currently stalled for lack of anyone working on it for H1. Chris
>>> Yeoh was the driving force in Grizzly on this, but he's hard at work on
>>> the Nova v3 API right now. Matt Treinish's going to pick this up on H2
>>> if no one else steps forward.
>> Robert - can you provide some help here? It seems like we're blocking on
>> some engineering challenges that I believe might be easy for you.
>>
>>> B currently has the issue that test criteria selection kind of sucks
>>> because of the lack of structure in the tempest tree. I'm currently
>>> working on this one to get it done by H1 -
>>> https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/tempest-repo-restructure,n,z
>>>
>>>
>>> C is currently probably a later in cycle task, just for time reasons.
>>>
>>> So B we should be able to do ~ H1 timeframe, we'll have the test suite
>>> in chunks that are easy to run bits and pieces of.
>>>
>>> A should be ~H2 (depending).
>>>
>>> But that's still a lot of weeks where we have test case contributors
>>> largely blocked because run time is an issue. Having a gate tag that we
>>> can annotate is just another lever. It also provides us with an excuse
>>> to get the multi tag support into the test cases, because developers
>>> really do want to run things like:
>> I agree, A and B are both important, and separate tasks.
>>
>> I don't agree that run time is an issue, actually, and if that is an
>> assertion we're operating under, I'd like to dig more in to why. The
>> tests have been running between 45 minutes and 1.5 hours for the past 6
>> months. I don't think that waiting another chunk of weeks is going to
>> kill us, and I don't think we should not accept new test cases because
>> of a run time limit. If we need to bump the gate timeout because the run
>> time is legitimately increased, that's not hard to do.
>>
>> If at all possible, I'd really like to get things sorted without gating
>> less.
>>
>>> ./run_tests.sh -t cinder  (or something similar)
>>>
>>> And get just the slice of tests that has a cinder interaction.
>>>
>>>       -Sean
>>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>


-- 
Sean Dague
http://dague.net