<div dir="ltr">Oh cool. I didn't realize it was deliberately limited already. I had assumed it was just hitting the resource limits for that queue. <div><br></div><div>So it looks like it's around 20 now. However, I would argue that shortening it more would help get patches through the gate. </div>
<div><br></div><div>For the sake of discussion, let's assume there is a 80% chance of success in one test run on a patch. So a given patch's probability of success is .8^n where n is the number of runs. </div>
<div><br></div><div>For the 1st patch in the queue, n is just one.</div><div>For the 2nd patch, n is 1 + the probability of a failure from patch 1.</div><div>For the 3rd patch, n is 1 + the probability of a failure in patch 2 or 1.</div>
<div>For the 4th patch, n is 1 + the probability of a failure in patch 3, 2, or 1.</div><div>...</div><div><br></div><div>Unfortunately my conditional probability skills are too shaky to trust an equation I come up with to represent the above scenario so I wrote a gate failure simulator [1].</div>
<div><br></div><div>At a queue size of 20 and an 80% success rate. The patch in position 20 only has a ~44% chance of getting merged.</div><div>However, with a queue size of 4, the patch in position 4 has a ~71% chance of getting merged.</div>
<div><br></div><div>You can try the simulator out yourself with various numbers. Maybe the odds of success are much better than 80% in one run and my point is moot, but I have several patches waiting to be merged that haven't made it through after ~3 tries each.</div>
<div><br></div><div><br></div><div>Cheers,</div><div>Kevin Benton</div><div><br></div><div>1. <a href="http://paste.openstack.org/show/83039/">http://paste.openstack.org/show/83039/</a></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 4:04 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="">On Thu, Jun 5, 2014 at 3:29 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Is it possible to make the depth of patches running tests in the gate very shallow during this high-probability of failure time? e.g. Allow only the top 4 to run tests and put the rest in the 'queued' state. Otherwise the already elevated probability of a patch failing is exacerbated by the fact that it gets retested every time a patch ahead of it in the queue fails. <div>
<br></div></div></blockquote></div><div>Such a good idea that we already do it.</div><div> </div><div><a href="http://status.openstack.org/zuul/" target="_blank">http://status.openstack.org/zuul/</a><br></div><div><br></div>
<div>The grey circles refer to patches that are in the queued state. But this only helps us from hitting resource starvation but doesn't help us get patches through the gate. We haven't been landing many patches this week [0]</div>
<div><br></div><div>[0] <a href="https://github.com/openstack/openstack/graphs/commit-activity" target="_blank">https://github.com/openstack/openstack/graphs/commit-activity</a></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div></div><div>--</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Thu, Jun 5, 2014 at 5:07 AM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>You may all have noticed things are really backed up in the gate right<br>
now, and you would be correct. (Top of gate is about 30 hrs, but if you<br>
do the math on ingress / egress rates the gate is probably really double<br>
that in transit time right now).<br>
<br>
We've hit another threshold where there are so many really small races<br>
in the gate that they are compounding to the point where fixing one is<br>
often failed by another one killing your job. This whole situation was<br>
exacerbated by the fact that while the transition from HP cloud 1.0 -><br>
1.1 was happening and we were under capacity, the check queue grew to<br>
500 with lots of stuff being approved.<br>
<br>
That flush all hit the gate at once. But it also means that those jobs<br>
passed in a very specific timing situation, which is different on the<br>
new HP cloud nodes. And the normal statistical distribution of some jobs<br>
on RAX and some on HP that shake out different races didn't happen.<br>
<br>
At this point we could really use help getting focus on only recheck<br>
bugs. The current list of bugs is here:<br>
<a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a><br>
<br>
Also our categorization rate is only 75% so there are probably at least<br>
2 critical bugs we don't even know about yet hiding in the failures.<br>
Helping categorize here -<br>
<a href="http://status.openstack.org/elastic-recheck/data/uncategorized.html" target="_blank">http://status.openstack.org/elastic-recheck/data/uncategorized.html</a><br>
would be handy.<br>
<br>
We're coordinating changes via an etherpad here -<br>
<a href="https://etherpad.openstack.org/p/gatetriage-june2014" target="_blank">https://etherpad.openstack.org/p/gatetriage-june2014</a><br>
<br>
If you want to help, jumping in #openstack-infra would be the place to go.<br>
<span><font color="#888888"><br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
</font></span><br></div></div><div>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</font></span></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>