<div dir="ltr">I kind of high jacked another thread with my testr problems, but I want to reiterate it directly on this one as they are my pain points I had from the transition.<div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">testr does not appear to support --exclusion[1] or --stop[2].</span><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">I have a work around for --exclusion, by:</div><blockquote style="font-family:arial,sans-serif;font-size:13px;margin:0px 0px 0px 40px;border:none;padding:0px">
<div>testr list-tests | egrep -v regex-exclude-list > unit-tests.txt</div><div>testr --load-list unit-tests.txt</div><div><br></div></blockquote><span style="font-family:arial,sans-serif;font-size:13px">I do not have a work around for --stop.</span><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">[1]<a href="https://bugs.launchpad.net/testrepository/+bug/1208610" target="_blank">https://bugs.launchpad.net/testrepository/+bug/1208610</a><br></div><div style="font-family:arial,sans-serif;font-size:13px">
[2]<a href="https://bugs.launchpad.net/testrepository/+bug/1211926" target="_blank">https://bugs.launchpad.net/testrepository/+bug/1211926</a></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 13, 2013 at 1:25 PM, Matthew Treinish <span dir="ltr"><<a href="mailto:mtreinish@kortar.org" target="_blank">mtreinish@kortar.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi everyone,<br>
<br>
So for the past month or so I've been working on getting tempest to work stably<br>
with testr in parallel. As part of this you may have noticed the testr-full<br>
jobs that get run on the zuul check queue. I was using that job to debug some<br>
of the more obvious race conditions and stability issues with running tempest<br>
in parallel. After a bunch of fixes to tempest and finding some real bugs in<br>
some of the projects things seem to have smoothed out.<br>
<br>
So I pushed the testr-full run to the gate queue earlier today. I'll be keeping<br>
track of the success rate of this job vs the serial job and use this as the<br>
determining factor before we push this live to be the default for all tempest<br>
runs. So assuming that the success rate matches up well enough with serial job<br>
on the gate queue then I will push out the change that will migrate all the<br>
voting jobs to run in parallel hopefully either Friday afternoon or early next<br>
week. Also, if anyone has any input on what threshold they feel is good enough<br>
for this I'd welcome any input on that. For example, do we want to ensure<br>
a >= 1:1 match for job success? Or would something like 90% as stable as the<br>
serial job be good enough considering the speed advantage. (The parallel runs<br>
take about half as much time as a full serial run, the parallel job normally<br>
finishes in ~25-30min) Since this affects almost every project I don't want to<br>
define this threshold without input from everyone.<br>
<br>
After there is some more data for the gate queue's parallel job I'll have some<br>
pretty graphite graphs that I can share comparing the success trends between<br>
the parallel and serial jobs.<br>
<br>
So at this point we're in the home stretch and I'm asking for everyone's help<br>
in getting this merged. So, if everyone who is reviewing and pushing commits<br>
could watch the results from these non-voting jobs and if things fail on the<br>
parallel job but not the serial job please investigate the failure and open a<br>
bug if necessary. If it turns out to be a bug in tempest please link it against<br>
this blueprint:<br>
<br>
<a href="https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest" target="_blank">https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest</a><br>
<br>
so that I'll give it the attention it deserves. I'd hate to get this close to<br>
getting this merged and have a bit of racy code get merged at the last second<br>
and block us for another week or two.<br>
<br>
I feel that we need to get this in before the H3 rush starts up as it will help<br>
everyone get through the extra review load faster.<br>
<br>
Thanks,<br>
<br>
Matt Treinish<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>