[openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

Kevin Benton blak111 at gmail.com
Fri Aug 28 12:34:42 UTC 2015


One of the patches that fixes one of the functional failures that has been
hitting is here: https://review.openstack.org/#/c/217927/

However, it failed in the DVR job on the 'test_router_rescheduling'
test.[1] This failure is because the logic to skip when DVR is enabled is
based on a check that will always return False.[2] I pushed a patch to
tempest to fix that [3] so once that gets merged we should be able to get
the one above merged.

For the py34 failures, they seem to have started around the same time as a
change was merged that adjusted the way they were ran so I proposed a
revert for that patch here: https://review.openstack.org/218244

1.
http://logs.openstack.org/27/217927/1/check/gate-tempest-dsvm-neutron-dvr/3361f9f/logs/testr_results.html.gz
2.
https://github.com/openstack/tempest/blob/8d827589e6589814e01089eb56b4d109274c781a/tempest/scenario/test_network_basic_ops.py#L662-L663
3. https://review.openstack.org/#/c/218242/

On Fri, Aug 28, 2015 at 4:09 AM, Sean Dague <sean at dague.net> wrote:

> We're at a 18hr backup in the gate, which is really unusual given the
> amount of decoupling. Even under our current load that means we're
> seeing huge failure rates causing resets.
>
> It appears one of the major culprits is the python34 tests in neutron,
> which were over a 40% failure rate recently - http://goo.gl/9wCerK
>
> That tends to lead to things like -
> http://dl.dropbox.com/u/6514884/screenshot_249.png - which means a huge
> amount of work has been reset. Right now 3 of 7 neutron patches in the
> gate that are within the sliding window are in a failure state (they are
> also the only current visible fails in the window).
>
> Looking at one of the patches in question -
> https://review.openstack.org/#/c/202207/ - shows it's been rechecked 3
> times, and these failures were seen in earlier runs.
>
> I do understand that people want to get their code merged, but
> rechecking patches that are failing this much without going after the
> root causes means everyone pays for it. This is blocking a lot of other
> projects from landing code in a timely manner.
>
> The functional tests seem to have a quite high failure rate as well from
> spot checking. If the results of these tests are mostly going to be
> ignored and rechecked, can we remove them from the gate definition on
> neutron so they aren't damaging the overall flow of the gate?
>
> Thanks,
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150828/674db78c/attachment.html>


More information about the OpenStack-dev mailing list