<div dir="ltr">Inline<div><br></div><div>Salvatore<br><div class="gmail_extra"><br><br><div class="gmail_quote">On 24 March 2014 23:01, Matthew Treinish <span dir="ltr"><<a href="mailto:mtreinish@kortar.org" target="_blank">mtreinish@kortar.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On Mon, Mar 24, 2014 at 09:56:09PM +0100, Salvatore Orlando wrote:<br>
> Thanks a lot!<br>
><br>
> We now need to get on these bugs, and define with QA an acceptable failure<br>
> rate criterion for switching the full job to voting.<br>
> It would be good to have a chance to only run the tests against code which<br>
> is already in master.<br>
> To this aim we might push a dummy patch, and keep it spinning in the check<br>
> queue.<br>
<br>
</div>Honestly, there isn't really a number. I had a thread trying to get consensus on<br>
that back when I first made tempest run in parallel. What I ended up doing back<br>
then and what we've done since for this kind of change is to just pick a slower<br>
week for the gate and just green light it, of course after checking to make sure<br>
if it blows up we're not blocking anything critical.</blockquote><div><br></div><div>Then I guess the ideal period would be after RC2s are cut.</div><div>Also, we'd need to run also a postgres flavour of the job at least.</div>
<div>Meaning that when calculating the probability of a patch passing in the gate is actually the combined probability of two jobs completing successfully.</div><div>On another note, we noticed that the duplicated jobs currently executed for redundancy in neutron actually seem to point all to the same build id.</div>
<div>I'm not sure then if we're actually executing each job twice or just duplicating lines in the jenkins report.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If it looks like it's<br>
passing at roughly the same rate as everything else and you guys think it's<br>
ready. 25% is definitely too high, for comparison when I looked at a couple of<br>
min. ago at the numbers for the past 4 days on the equivalent job with<br>
nova-network it only failed 4% of the time. (12 out of 300) But that number does<br>
fluctuate quite a bit for example looking at the past week the number grows to<br>
11.6%. (171 out of 1480)</blockquote><div><br></div><div>Even with 11.6% I would not enable it.</div><div>Running mysql and pg jobs this will give us a combined success rate of 78.1%, which pretty much means the chances of clearing successfully a 5-deep queue in the gate will be a mere 29%. My "gut" metric is that we should achieve a degree of pass rate which allows us to clear a 10-deep gate queue with a 50% success rate. This translates to a 3.5% failure rate per job, which is indeed inline with what's currently observed for nova-network.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Doing it this way doesn't seem like the best, but until it's gating things<br>
really don't get the attention they deserve and more bugs will just slip in<br>
while you wait. There will most likely be initial pain after it merges, but it's<br>
the only real way to lock it down and make forward progress.<br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
-Matt Treinish<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
> On 24 March 2014 21:45, Rossella Sblendido <<a href="mailto:rsblendido@suse.com">rsblendido@suse.com</a>> wrote:<br>
><br>
> > Hello all,<br>
> ><br>
> > here is an update regarding the Neutron full parallel job.<br>
> > I used the following Logstash query [1] that checks the failures of the<br>
> > last<br>
> > 4 days (the last bug fix related with the full job was merged 4 days ago).<br>
> > These are the results:<br>
> ><br>
> > 123 failure (25% of the total)<br>
> ><br>
> > I took a sample of 50 failures and I obtained the following:<br>
> ><br>
> > 22% legitimate failures (they are due to the code change introduced by the<br>
> > patch)<br>
> > 22% infra issues<br>
> > 12% <a href="https://bugs.launchpad.net/openstack-ci/+bug/1291611" target="_blank">https://bugs.launchpad.net/openstack-ci/+bug/1291611</a><br>
> > 12% <a href="https://bugs.launchpad.net/tempest/+bug/1281969" target="_blank">https://bugs.launchpad.net/tempest/+bug/1281969</a><br>
> > 8% <a href="https://bugs.launchpad.net/tempest/+bug/1294603" target="_blank">https://bugs.launchpad.net/tempest/+bug/1294603</a><br>
> > 3% <a href="https://bugs.launchpad.net/neutron/+bug/1283522" target="_blank">https://bugs.launchpad.net/neutron/+bug/1283522</a><br>
> > 3% <a href="https://bugs.launchpad.net/neutron/+bug/1291920" target="_blank">https://bugs.launchpad.net/neutron/+bug/1291920</a><br>
> > 3% <a href="https://bugs.launchpad.net/nova/+bug/1290642" target="_blank">https://bugs.launchpad.net/nova/+bug/1290642</a><br>
> > 3% <a href="https://bugs.launchpad.net/tempest/+bug/1252971" target="_blank">https://bugs.launchpad.net/tempest/+bug/1252971</a><br>
> > 3% <a href="https://bugs.launchpad.net/horizon/+bug/1257885" target="_blank">https://bugs.launchpad.net/horizon/+bug/1257885</a><br>
> > 3% <a href="https://bugs.launchpad.net/tempest/+bug/1292242" target="_blank">https://bugs.launchpad.net/tempest/+bug/1292242</a><br>
> > 3% <a href="https://bugs.launchpad.net/neutron/+bug/1277439" target="_blank">https://bugs.launchpad.net/neutron/+bug/1277439</a><br>
> > 3% <a href="https://bugs.launchpad.net/neutron/+bug/1283599" target="_blank">https://bugs.launchpad.net/neutron/+bug/1283599</a><br>
> ><br>
> > cheers,<br>
> ><br>
> > Rossella<br>
> ><br>
> > [1] <a href="http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOi" target="_blank">http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOi</a><br>
> > BcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWZ1bGxcIiBBTkQgbWVzc2<br>
> > FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIgQU5EIHRhZ3M6Y29uc29sZSIsIm<br>
> > ZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiY3VzdG9tIiwiZ3<br>
> > JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7ImZyb20iOiIyMDE0LTAzLTIwVD<br>
> > EzOjU0OjI1KzAwOjAwIiwidG8iOiIyMDE0LTAzLTI0VDEzOjU0OjI1KzAwOj<br>
> > AwIiwidXNlcl9pbnRlcnZhbCI6IjAifSwibW9kZSI6IiIsImFuYWx5emVfZm<br>
> > llbGQiOiIiLCJzdGFtcCI6MTM5NTY3MDY2ODc0OX0=<br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
<br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div></div>