<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 3:05 PM, Kyle Mestery <span dir="ltr"><<a href="mailto:mestery@noironetworks.com" target="_blank">mestery@noironetworks.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Thu, Jun 5, 2014 at 7:07 AM, Sean Dague <<a href="mailto:sean@dague.net">sean@dague.net</a>> wrote:<br>
> You may all have noticed things are really backed up in the gate right<br>
> now, and you would be correct. (Top of gate is about 30 hrs, but if you<br>
> do the math on ingress / egress rates the gate is probably really double<br>
> that in transit time right now).<br>
><br>
> We've hit another threshold where there are so many really small races<br>
> in the gate that they are compounding to the point where fixing one is<br>
> often failed by another one killing your job. This whole situation was<br>
> exacerbated by the fact that while the transition from HP cloud 1.0 -><br>
> 1.1 was happening and we were under capacity, the check queue grew to<br>
> 500 with lots of stuff being approved.<br>
><br>
> That flush all hit the gate at once. But it also means that those jobs<br>
> passed in a very specific timing situation, which is different on the<br>
> new HP cloud nodes. And the normal statistical distribution of some jobs<br>
> on RAX and some on HP that shake out different races didn't happen.<br>
><br>
> At this point we could really use help getting focus on only recheck<br>
> bugs. The current list of bugs is here:<br>
> <a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a><br>
><br>
> Also our categorization rate is only 75% so there are probably at least<br>
> 2 critical bugs we don't even know about yet hiding in the failures.<br>
> Helping categorize here -<br>
> <a href="http://status.openstack.org/elastic-recheck/data/uncategorized.html" target="_blank">http://status.openstack.org/elastic-recheck/data/uncategorized.html</a><br>
> would be handy.<br>
><br>
> We're coordinating changes via an etherpad here -<br>
> <a href="https://etherpad.openstack.org/p/gatetriage-june2014" target="_blank">https://etherpad.openstack.org/p/gatetriage-june2014</a><br>
><br>
> If you want to help, jumping in #openstack-infra would be the place to go.<br>
><br>
</div></div>For the Neutron "ssh timeout" issue [1], we think we know why it's<br>
spiked recently. This tempest change [2] may have made the situation<br>
worse. We'd like to propose reverting that change with the review here<br>
[3], at which point we can resubmit it and continue debugging this.<br>
But this should help relieve the pressure caused by the recent surge<br>
in this bug.<br>
<br>
Does this sound like a workable plan to get things moving again?<br></blockquote><div><br></div><div><br></div><div>As we discussed on IRC yes, and thank you for hunting this one down.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
Kyle<br>
<br>
[1] <a href="https://bugs.launchpad.net/bugs/1323658" target="_blank">https://bugs.launchpad.net/bugs/1323658</a><br>
[2] <a href="https://review.openstack.org/#/c/90427/" target="_blank">https://review.openstack.org/#/c/90427/</a><br>
[3] <a href="https://review.openstack.org/#/c/97245/" target="_blank">https://review.openstack.org/#/c/97245/</a><br>
<div class="im HOEnZb"><br>
> -Sean<br>
><br>
> --<br>
> Sean Dague<br>
> <a href="http://dague.net" target="_blank">http://dague.net</a><br>
><br>
><br>
</div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>