Open Stack

On 08/13/2013 04:25 PM, Matthew Treinish wrote:
>
> Hi everyone,
>
> So for the past month or so I've been working on getting tempest to work stably
> with testr in parallel. As part of this you may have noticed the testr-full
> jobs that get run on the zuul check queue. I was using that job to debug some
> of the more obvious race conditions and stability issues with running tempest
> in parallel. After a bunch of fixes to tempest and finding some real bugs in
> some of the projects things seem to have smoothed out.
>
> So I pushed the testr-full run to the gate queue earlier today. I'll be keeping
> track of the success rate of this job vs the serial job and use this as the
> determining factor before we push this live to be the default for all tempest
> runs. So assuming that the success rate matches up well enough with serial job
> on the gate queue then I will push out the change that will migrate all the
> voting jobs to run in parallel hopefully either Friday afternoon or early next
> week. Also, if anyone has any input on what threshold they feel is good enough
> for this I'd welcome any input on that. For example, do we want to ensure
> a >= 1:1 match for job success? Or would something like 90% as stable as the
> serial job be good enough considering the speed advantage. (The parallel runs
> take about half as much time as a full serial run, the parallel job normally
> finishes in ~25-30min) Since this affects almost every project I don't want to
> define this threshold without input from everyone.
>
> After there is some more data for the gate queue's parallel job I'll have some
> pretty graphite graphs that I can share comparing the success trends between
> the parallel and serial jobs.
>
> So at this point we're in the home stretch and I'm asking for everyone's help
> in getting this merged. So, if everyone who is reviewing and pushing commits
> could watch the results from these non-voting jobs and if things fail on the
> parallel job but not the serial job please investigate the failure and open a
> bug if necessary. If it turns out to be a bug in tempest please link it against
> this blueprint:
>
> https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
>
> so that I'll give it the attention it deserves. I'd hate to get this close to
> getting this merged and have a bit of racy code get merged at the last second
> and block us for another week or two.
>
> I feel that we need to get this in before the H3 rush starts up as it will help
> everyone get through the extra review load faster.

Fantastic work on this, Matthew. Appreciate the effort immensely.

Best,
-jay