<div dir="ltr">I gave a -2 yesterday to all my Neutron patches. I did that because I thought something was wrong with them, but then I started to realize it's a general problem.<div>It makes sense to give some priority to the patches Eugene linked, even if it would be better to have some people root causing the issues that are plaguing the neutron gate. I saw at least 4 different failure modes and I'm not even sure whether it's due to Neutron or the tempest changes we merged during the sprint.</div>
<div><br></div><div>Salvatore</div><div><br></div><div>PS: I can't look at those issues today, as I have some backlog from last week to deal with.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 20 January 2014 09:43, Eugene Nikanorov <span dir="ltr"><<a href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Sean,<div><br></div><div>I think the following 2 commits in neutron are essential for bringing neutron jobs back to acceptable level of failure rate:</div>
<div><a href="https://review.openstack.org/#/c/67537/" target="_blank">https://review.openstack.org/#/c/67537/</a><br>
</div><div><a href="https://review.openstack.org/#/c/66670/" target="_blank">https://review.openstack.org/#/c/66670/</a><br></div><div><br></div><div>Thanks,</div><div>Eugene.</div></div><div class="gmail_extra"><br><br>
<div class="gmail_quote"><div><div class="h5">
On Mon, Jan 20, 2014 at 6:33 PM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div class="h5">
Anyone that's looked at the gate this morning... knows things aren't<br>
good. It turns out that a few new races got into OpenStack last week,<br>
which are causing a ton of pain, and have put us dramatically over the edge.<br>
<br>
We've not tracked down all of them, but 2 that are quite important to<br>
address are:<br>
<br>
- Bug 1270680 - v3 extensions api inherently racey wrt instances<br>
- Bug 1270608 - n-cpu 'iSCSI device not found' log causes<br>
gate-tempest-dsvm-*-full to fail<br>
<br>
Both can be seen as very new issues here -<br>
<a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a><br>
<br>
We've got a short term work around on 1270680 which we're going to take<br>
into the gate now (and fix it better later).<br>
<br>
1270608 is still in desperate need of fixing.<br>
<br>
<br>
Neutron is in a whole other level of pain. Over the weekend I found the<br>
isolated jobs are in a 70% fail state, which means the overall chance<br>
for success for Neutron / Neutron client patches are < 5%. As such I'd<br>
suggest a moritorium for them going into the gate at this point, as they<br>
are basically guarunteed to fail.<br>
<span><font color="#888888"><br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
Samsung Research America<br>
<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a> / <a href="mailto:sean.dague@samsung.com" target="_blank">sean.dague@samsung.com</a><br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
</font></span><br></div></div>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>