<div dir="ltr">One of the patches that fixes one of the functional failures that has been hitting is here: <a href="https://review.openstack.org/#/c/217927/">https://review.openstack.org/#/c/217927/</a><div><br></div><div>However, it failed in the DVR job on the 'test_router_rescheduling' test.[1] This failure is because the logic to skip when DVR is enabled is based on a check that will always return False.[2] I pushed a patch to tempest to fix that [3] so once that gets merged we should be able to get the one above merged.</div><div><br></div><div>For the py34 failures, they seem to have started around the same time as a change was merged that adjusted the way they were ran so I proposed a revert for that patch here: <a href="https://review.openstack.org/218244">https://review.openstack.org/218244</a></div><div><br></div><div>1. <a href="http://logs.openstack.org/27/217927/1/check/gate-tempest-dsvm-neutron-dvr/3361f9f/logs/testr_results.html.gz">http://logs.openstack.org/27/217927/1/check/gate-tempest-dsvm-neutron-dvr/3361f9f/logs/testr_results.html.gz</a></div><div>2. <a href="https://github.com/openstack/tempest/blob/8d827589e6589814e01089eb56b4d109274c781a/tempest/scenario/test_network_basic_ops.py#L662-L663">https://github.com/openstack/tempest/blob/8d827589e6589814e01089eb56b4d109274c781a/tempest/scenario/test_network_basic_ops.py#L662-L663</a></div><div>3. <a href="https://review.openstack.org/#/c/218242/">https://review.openstack.org/#/c/218242/</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 28, 2015 at 4:09 AM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We're at a 18hr backup in the gate, which is really unusual given the<br>
amount of decoupling. Even under our current load that means we're<br>
seeing huge failure rates causing resets.<br>
<br>
It appears one of the major culprits is the python34 tests in neutron,<br>
which were over a 40% failure rate recently - <a href="http://goo.gl/9wCerK" rel="noreferrer" target="_blank">http://goo.gl/9wCerK</a><br>
<br>
That tends to lead to things like -<br>
<a href="http://dl.dropbox.com/u/6514884/screenshot_249.png" rel="noreferrer" target="_blank">http://dl.dropbox.com/u/6514884/screenshot_249.png</a> - which means a huge<br>
amount of work has been reset. Right now 3 of 7 neutron patches in the<br>
gate that are within the sliding window are in a failure state (they are<br>
also the only current visible fails in the window).<br>
<br>
Looking at one of the patches in question -<br>
<a href="https://review.openstack.org/#/c/202207/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/202207/</a> - shows it's been rechecked 3<br>
times, and these failures were seen in earlier runs.<br>
<br>
I do understand that people want to get their code merged, but<br>
rechecking patches that are failing this much without going after the<br>
root causes means everyone pays for it. This is blocking a lot of other<br>
projects from landing code in a timely manner.<br>
<br>
The functional tests seem to have a quite high failure rate as well from<br>
spot checking. If the results of these tests are mostly going to be<br>
ignored and rechecked, can we remove them from the gate definition on<br>
neutron so they aren't damaging the overall flow of the gate?<br>
<br>
Thanks,<br>
<br>
-Sean<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Sean Dague<br>
<a href="http://dague.net" rel="noreferrer" target="_blank">http://dague.net</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>