<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 24 August 2016 at 10:30, Armando M. <span dir="ltr"><<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On 22 August 2016 at 18:25, Armando M. <span dir="ltr"><<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On 22 August 2016 at 09:51, Armando M. <span dir="ltr"><<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Neutrinos,<div><br></div><div>Since the merge of test [1], a race has surfaced, which leads to failure as reported in [2]. A number of Neutron's jobs are affected, and even though the failure is only intermittent, it is particularly bad for the linux bridge job.</div><div><br></div><div>If I can ask you to hold your +2/+W until [3] puts the fire out, zuul will be grateful.</div></div></blockquote><div><br></div></span><div>An update:</div><div><br></div><div>We have [1] in the gate queue, and Kevin is going full steam with [2], which will mitigate the other intermittent issues we're facing in functional and unit tests. Please bear with us.</div><div><br></div><div>Cheers,</div><div>Armando</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/358753/" target="_blank">https://review.openstack.org/#<wbr>/c/358753/</a></div><div>[2] <a href="https://review.openstack.org/#/c/356530/" target="_blank">https://review.openstack.o<wbr>rg/#/c/356530/</a></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Thanks,</div><div>Armando</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/327191/" target="_blank">https://review.openstack.o<wbr>rg/#/c/327191/</a></div><div>[2] <a href="https://bugs.launchpad.net/neutron/+bug/1615710" target="_blank">https://bugs.launchpad.net<wbr>/neutron/+bug/1615710</a></div><div>[3] <a href="https://review.openstack.org/#/c/358753/" target="_blank">https://review.openstack.o<wbr>rg/#/c/358753/</a></div></div></blockquote></span></div></div></div></blockquote><div><br></div><div><br></div></div></div><div>One more update:</div><div><br></div><div>We're not quite out of the woods, yet so approve/recheck gently please.</div><div><br></div><div>Things have become slightly better but we still have a pending fix for Neutron [1]. Recently we also observed zuul nodes dying unexpectedly on the OSIC cloud (that runs Neutron on IPv6) and that had an impact on the overall sluggishness of the gate [2]. Once all of the fixes are in, we should be back in business...until the next fire!</div></div></div></div></blockquote><div><br></div><div>Another (final) update:</div><div><br></div><div>Changes [1] have merged. That should take care of the infra node resets.</div><div><br></div><div>At this point, there's [2] to wrap up, but things look largely recovered [3,4] (though we need [5] to fully see a current picture). Please keep an eye on [6], and help report/address critical fixes in the busy times ahead.</div><div><br></div><div>Cheers,</div><div>Armando</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/q/Ia044fff2a1731ab6c04f82aea47096b425e0c0a0,n,z">https://review.openstack.org/#/q/Ia044fff2a1731ab6c04f82aea47096b425e0c0a0,n,z</a></div><div>[2] <a href="https://review.openstack.org/#/c/356530/">https://review.openstack.org/#/c/356530/</a></div><div>[3] <a href="http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=3&fullscreen">http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=3&fullscreen</a></div><div>[4] <a href="http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=5&fullscreen">http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=5&fullscreen</a></div><div>[5] <a href="https://review.openstack.org/#/c/358462/">https://review.openstack.org/#/c/358462/</a></div><div>[6] <a href="https://bugs.launchpad.net/neutron/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.tag=gate-failure">https://bugs.launchpad.net/neutron/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.tag=gate-failure</a></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Cheers,</div><div>Armando</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/356530/" target="_blank">https://review.openstack.<wbr>org/#/c/356530/</a></div><div>[2] <a href="https://bugs.launchpad.net/neutron/+bug/1616282" target="_blank">https://bugs.launchpad.net/<wbr>neutron/+bug/1616282</a> <br></div></div><br></div></div>
</blockquote></div><br></div></div>