<div dir="ltr"><div>TL;DR: Gate is failing 23% of the time due to bugs in nova, neutron and tempest. We need help fixing these bugs.<br></div><div><br></div><div><br></div>Hi All,<div><br></div><div>Before going any further we have a bug that is affecting gate and stable, so its getting top priority here. elastic-recheck currently doesn't track unit tests because we don't expect them to fail very often. Turns out that assessment was wrong, we now have a nova py27 unit test bug in gate and stable gate.</div>
<div><div><br></div><div><a href="https://bugs.launchpad.net/nova/+bug/1216851">https://bugs.launchpad.net/nova/+bug/1216851</a></div><div>Title: nova unit tests occasionally fail migration tests for mysql and postgres</div>
</div><div>Hits</div><div> FAILURE: 74</div><div>The failures appear multiple times for a single job, and some of those are due to bad patches in the check queue. But this is being seen in stable and trunk gate so something is definitely wrong.</div>
<div><br></div><div>=======</div><div><br></div><div><br></div><div>Its time for another edition of of 'Top Gate Bugs.' I am sending this out now because in addition to our usual gate bugs a few new ones have cropped up recently, and as we saw a few weeks ago it doesn't take very many new bugs to wedge the gate.</div>
<div><br></div><div>Currently the gate has a failure rate of at least 23%! [0]</div><div><br></div><div>Note: this email was generated with <a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a> and 'elastic-recheck-success' [1]</div>
<div><br></div><div>1) <a href="https://bugs.launchpad.net/bugs/1253896" target="_blank">https://bugs.launchpad.net/bugs/1253896</a></div><div>Title: test_minimum_basic_scenario fails with SSHException: Error reading SSH protocol banner</div>
<div>Projects: neutron, nova, tempest</div><div>Hits</div><div> FAILURE: 324</div><div>This one has been around for several weeks now and although we have made some attempts at fixing this, we aren't any closer at resolving this then we were a few weeks ago.</div>
<div><br></div><div>2) <a href="https://bugs.launchpad.net/bugs/1251448" target="_blank">https://bugs.launchpad.net/bugs/1251448</a></div><div>Title: BadRequest: Multiple possible networks found, use a Network ID to be more specific. </div>
<div>Project: neutron</div><div>Hits</div><div> FAILURE: 141</div><div><br></div><div>3) <a href="https://bugs.launchpad.net/bugs/1249065" target="_blank">https://bugs.launchpad.net/bugs/1249065</a></div><div>Title: Tempest failure: tempest/scenario/test_snapshot_pattern.py</div>
<div>Project: nova</div><div>Hits</div><div> FAILURE: 112</div><div>This is a bug in nova's neutron code.</div><div><br></div><div>4) <a href="https://bugs.launchpad.net/bugs/1250168">https://bugs.launchpad.net/bugs/1250168</a></div>
<div>Title: gate-tempest-devstack-vm-neutron-large-ops is failing</div><div>Projects: neutron, nova</div><div>Hits</div><div> FAILURE: 94</div><div>This is an old bug that was fixed, but came back on December 3rd. So this is a recent regression. This may be an infra issue.</div>
<div><br></div><div>5) <a href="https://bugs.launchpad.net/bugs/1210483">https://bugs.launchpad.net/bugs/1210483</a></div><div>Title: ServerAddressesTestXML.test_list_server_addresses FAIL</div><div>Projects: neutron, nova</div>
<div>Hits</div><div> FAILURE: 73</div><div>This has had some attempts made at fixing it but its still around.</div><div><br></div><div><br></div><div>In addition to the existing bugs, we have some new bugs on the rise:</div>
<div><br></div><div><div>1) <a href="https://bugs.launchpad.net/bugs/1257626">https://bugs.launchpad.net/bugs/1257626</a></div><div>Title: Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"</div>
<div>Project: nova</div><div>Hits</div><div> FAILURE: 52</div><div>large-ops only bug. This has been around for at least two weeks, but we have seen this in higher numbers starting around December 3rd. This may be an infrastructure issue as the neutron-large-ops started failing more around the same time.</div>
<div><br></div><div>2) <a href="https://bugs.launchpad.net/bugs/1257641">https://bugs.launchpad.net/bugs/1257641</a></div><div>Title: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances</div><div>
Projects: nova, tempest</div><div>Hits</div><div> FAILURE: 41</div></div><div>Like the previous bug, this has been around for at least two weeks but appears to be on the rise.</div><div><br></div><div><br></div><div><br>
</div><div>Raw Data: <a href="http://paste.openstack.org/show/54419/" target="_blank">http://paste.openstack.org/show/54419/</a></div><div><br></div><div><br></div><div>best,</div><div>Joe</div><div><br></div><div>
<br></div><div>[0] failure rate = 1-(success rate gate-tempest-dsvm-neutron)*(success rate ...) * ...</div><div><br></div><div><div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-tempest-dsvm-neutron = 0.00</span><br>
</div><div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-tempest-dsvm-neutron-large-ops = 11.11</span><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br></span></div>
<div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-tempest-dsvm-full = 11.11</span><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br></span></div><div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-tempest-dsvm-large-ops = 4.55</span><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br>
</span></div><div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-tempest-dsvm-postgres-full = 10.00</span><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br></span></div>
<div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px">gate-grenade-dsvm = 0.00</span></div></div><div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br></span></div><div><font color="#545454" face="sans-serif">(I hope I got the math right here)</font></div>
<div><span style="color:rgb(84,84,84);font-family:sans-serif;font-size:13px"><br></span></div><div>[1] <a href="http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/cmd/check_success.py" target="_blank">http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/cmd/check_success.py</a></div>
</div>