<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Ciao Salvatore,<br>
<br>
thanks a lot for analyzing the failures!<br>
<br>
This link is not working for me:<br>
7) <a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/neutron/+bug/1253533">https://bugs.launchpad.net/neutron/+bug/1253533</a><br>
<br>
I took a minor bug that was not assigned. Most of the bugs are
assigned to you, I was wondering if you´d use some help. I guess we
can coordinate better when you are online.<br>
<br>
cheers,<br>
<br>
Rossella<br>
<br>
<div class="moz-cite-prefix">On 02/23/2014 03:14 AM, Salvatore
Orlando wrote:<br>
</div>
<blockquote
cite="mid:CAGR=i3j6z18=9yAaCS7znjR4rZfOu7ZOMKj2zySXG2c0Q80ddQ@mail.gmail.com"
type="cite">
<div dir="ltr">I have tried to collect more information on neutron
full job failures.
<div><br>
</div>
<div>So far there have been 219 failures and 891 successes, for
an overall success rate of 19.8% which is inline with Sean's
evaluation.</div>
<div>The count has performed exclusively on jobs executed
against master branch. The failure rate for stable/havana is
higher; indeed the job there still triggers bug 1273386 as it
performs nbd mounting, and several fixes for the l2/l3 agents
were not backported (or not backportable).</div>
<div><br>
</div>
<div>It is worth noting that actually some of the failures were
because of infra issues. Unfortunately, it is not obvious to
me how to define a logstash query for that. Nevertheless, it
will be better to err on the side of safety and estimate
failure rate to be about 20%.</div>
<div><br>
</div>
<div>I did then a classification of 63 failures, finding out the
following:</div>
<div>- 25 failures were for infra issues, 1 failure was due to a
flaw in a patch, leaving 37 "real" failures to analyse</div>
<div> * In the same timeframe 203 jobs succeeded, giving a
potential failure rate after excluding infra issues of 15.7%</div>
<div>- 2 bugs were responsible for 25 of these 37 failures</div>
<div> * they are the "SSH protocol banner issue", and the
well-knows DB lock timeouts</div>
<div>- bug 1253896 (the infamous SSH timeout bug) was hit only
twice. The elastic recheck count is much higher because
failures for the SSH protocol banner error (1265495) are being
classified as bug 1253896.</div>
<div> * actually in the past 48 hours only 2 voting neutron
jobs hit this failure. This is probably a great improvement
compared with a few weeks ago.</div>
<div>- Some failures are due to bug already known and tracked,
other failures are due to bugs either unforeseen so far or not
tracked. In the latter case a bug report has been filed.</div>
<div><br>
</div>
<div>It seems therefore that there are two high priority bugs to
address:</div>
<div>1) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1283522">https://bugs.launchpad.net/neutron/+bug/1283522</a>
(16 occurrences, 43.2% of failure, 6.67% globally)<br>
</div>
<div> * Check whether we can resume the split between API
server and RPC server discussion)</div>
<div>2) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1265495">https://bugs.launchpad.net/neutron/+bug/1265495</a>
(9/37 = 24.3% of failures, 3.75% globally)<br>
</div>
<div><br>
</div>
<div>And several minor bugs (affecting tempest and/or neutron)</div>
<div>Each one of the following bugs was found no more than twice
in our analysis:</div>
<div>3) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1254890">https://bugs.launchpad.net/neutron/+bug/1254890</a>
(possibly a nova bug, but it hit the neutron full job once)</div>
<div>4) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1283599">https://bugs.launchpad.net/neutron/+bug/1283599</a></div>
<div>5) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1277439">https://bugs.launchpad.net/neutron/+bug/1277439</a></div>
<div>6) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1253896">https://bugs.launchpad.net/neutron/+bug/1253896</a></div>
<div>7) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1253533">https://bugs.launchpad.net/neutron/+bug/1253533</a><br>
</div>
<div>8) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/tempest/+bug/1283535">https://bugs.launchpad.net/tempest/+bug/1283535</a>
(possibly not a neutron bug)</div>
<div>9) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/tempest/+bug/1253993">https://bugs.launchpad.net/tempest/+bug/1253993</a>
(need to devise new solutions for improving agent loop times)</div>
<div> * there is already a patch under review for bulking
device details requests</div>
<div>10) <a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1283518">https://bugs.launchpad.net/neutron/+bug/1283518</a></div>
<div><br>
</div>
<div>In my humble opinion, it is therefore important to have
immediately a plan for ensuring bugs #1 and #2 are solved or
at least consistently mitigated by icehouse. It would also be
good to identify assignees for bug #3 to bug #10.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Salvatore</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 21 February 2014 14:44, Sean Dague <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Yesterday
during the QA meeting we realized that the neutron full job,<br>
which includes tenant isolation, and full parallelism, was
passing quite<br>
often in the experimental queue. Which was actually news to
most of us,<br>
as no one had been keeping a close eye on it.<br>
<br>
I moved that to a non-voting job on all projects. A spot
check overnight<br>
is that it's failing about twice as often as the regular
neutron job.<br>
Which is too high a failure rate to make it voting, but it's
close.<br>
<br>
This would be the time for a final hard push by the neutron
team to get<br>
to the bottom of these failures to bring the pass rate to
the level of<br>
the existing neutron job, then we could make neutron full
voting.<br>
<br>
This is a *huge* move forward from where things were at the
Havana<br>
summit. I want to thank the Neutron team for getting so
aggressive about<br>
getting this testing working. I was skeptical we could get
there within<br>
the cycle, but a last push could actually get us neutron
parity in the<br>
gate by i3.<br>
<span class="HOEnZb"><font color="#888888"><br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
Samsung Research America<br>
<a moz-do-not-send="true" href="mailto:sean@dague.net">sean@dague.net</a>
/ <a moz-do-not-send="true"
href="mailto:sean.dague@samsung.com">sean.dague@samsung.com</a><br>
<a moz-do-not-send="true" href="http://dague.net"
target="_blank">http://dague.net</a><br>
<br>
</font></span><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>