<div dir="ltr">Bug <font face="arial, sans-serif">1294603 has the root cause in LBaaS, which should be fixed by <a href="https://review.openstack.org/#/c/81537/">https://review.openstack.org/#/c/81537/</a></font><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif">Thanks,</font></div><div><font face="arial, sans-serif">Eugene. </font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 28, 2014 at 7:29 PM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=""><br>
<br>
On 3/27/2014 8:00 AM, Salvatore Orlando wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
<br>
On 26 March 2014 19:19, James E. Blair <<a href="mailto:jeblair@openstack.org" target="_blank">jeblair@openstack.org</a><br></div>
<mailto:<a href="mailto:jeblair@openstack.org" target="_blank">jeblair@openstack.org</a>><u></u>> wrote:<br>
<br>
Salvatore Orlando <<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a> <mailto:<a href="mailto:sorlando@nicira.com" target="_blank">sorlando@nicira.com</a>>><div><div class="h5"><br>
writes:<br>
<br>
> On another note, we noticed that the duplicated jobs currently<br>
executed for<br>
> redundancy in neutron actually seem to point all to the same<br>
build id.<br>
> I'm not sure then if we're actually executing each job twice or just<br>
> duplicating lines in the jenkins report.<br>
<br>
Thanks for catching that, and I'm sorry that didn't work right. Zuul is<br>
in fact running the jobs twice, but it is only looking at one of them<br>
when sending reports and (more importantly) decided whether the change<br>
has succeeded or failed. Fixing this is possible, of course, but turns<br>
out to be a rather complicated change. Since we don't make heavy use of<br>
this feature, I lean toward simply instantiating multiple instances of<br>
identically configured jobs and invoking them (eg "neutron-pg-1",<br>
"neutron-pg-2").<br>
<br>
Matthew Treinish has already worked up a patch to do that, and I've<br>
written a patch to revert the incomplete feature from Zuul.<br>
<br>
<br>
That makes sense to me. I think it is just a matter about the results<br>
are reported to gerrit since from what I gather in logstash the jobs are<br>
executed twice for each new patchset or recheck.<br>
<br>
<br>
For the status of the full job, I gave a look at the numbers reported by<br>
Rossella.<br>
All the bugs are already known; some of them are not even bug; others<br>
have been recently fixed (given the time span of Rossella analysis and<br>
the fact it covers also non-rebased patches it might be possible to have<br>
this kind of false positive).<br>
<br>
of all full job failures, 44% should be discarded.<br>
Bug 1291611 (12%) is definitely not a neutron bug... hopefully.<br>
Bug 1281969 (12%) is really too generic.<br>
It bears the hallmark of bug1283522, and therefore the high number might<br>
be due to the fact that trunk was plagued by this bug up to a few days<br>
before the analysis.<br>
However, it's worth noting that there is also another instance of "lock<br>
timeout" which has caused 11 failures in full job in the past week.<br>
A new bug has been filed for this issue:<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1298355" target="_blank">https://bugs.launchpad.net/<u></u>neutron/+bug/1298355</a><br>
Bug 1294603 was related to a test now skipped. It is still being debated<br>
whether the problem lies in test design, neutron LBaaS or neutron L3.<br>
<br>
The following bugs seem not to be neutron bugs:<br>
1290642, 1291920, 1252971, 1257885<br>
<br>
Bug 1292242 appears to have been fixed while the analysis was going on<br>
Bug 1277439 instead is already known to affects neutron jobs occasionally.<br>
<br>
The actual state of the job is perhaps better than what the raw numbers<br>
say. I would keep monitoring it, and then make it voting after the<br>
Icehouse release is cut, so that we'll be able to deal with possible<br>
higher failure rate in the "quiet" period of the release cycle.<br>
<br>
<br>
<br>
-Jim<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br></div></div>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><div class=""><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</div></blockquote>
<br>
I reported this bug [1] yesterday. This was hit in our internal Tempest runs on RHEL 6.5 with x86_64 and the nova libvirt driver with the neutron openvswitch ML2 driver. We're running without tenant isolation on python 2.6 (no testr yet) so the tests are in serial. We're running basically the full API/CLI/Scenarios tests though, no filtering on the smoke tag.<br>
<br>
Out of 1,971 tests run, we had 3 failures where a nova instance failed to spawn because networking callback events failed, i.e. neutron sends a server event request to nova and it's a bad URL so nova API pukes and then the networking request in neutron server fails. As linked in the bug report I'm seeing the same neutron server log error showing up in logstash for community jobs but it's not 100% failure. I haven't seen the n-api log error show up in logstash though.<br>
<br>
Just bringing this to people's attention in case anyone else sees it.<br>
<br>
[1] <a href="https://bugs.launchpad.net/nova/+bug/1298640" target="_blank">https://bugs.launchpad.net/<u></u>nova/+bug/1298640</a><span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div>