[openstack-dev] [neutron] 40% failure on neutron python3.4 tests in the gate

Assaf Muller amuller at redhat.com
Fri Aug 28 19:21:32 UTC 2015


On Fri, Aug 28, 2015 at 1:50 PM, Salvatore Orlando <salv.orlando at gmail.com>
wrote:

>
>
> On 28 August 2015 at 16:57, Sean Dague <sean at dague.net> wrote:
>
>> On 08/28/2015 11:20 AM, Assaf Muller wrote:
>> > To recap, we had three issues impacting the gate queue:
>> >
>> > 1) The neutron functional job has had a high failure rate for a while
>> > now. Since it's impacting the gate,
>> > I've removed it from the gate queue but kept it in the Neutron check
>> queue:
>> > https://review.openstack.org/#/c/218302/
>> >
>> > If you'd like to help, the the list of bugs impacting the Neutron
>> > functional job is linked in that patch.
>> >
>> > 2) A new Tempest scenario test was added that caused the DVR job failure
>> > rate to sky rocket to over 50%.
>> > It actually highlighted a legit bug with DVR and legacy routers. Kevin
>> > proposed a patch that skips that test
>> > entirely until we can resolve the bug in Neutron:
>> > https://review.openstack.org/#/c/218242/ (Currently it tries to skip
>> the
>> > test conditionally, the next PS will skip the test entirely).
>> >
>> > 3) The Neutron py34 job has been made unstable due to a recent change
>> > (By me, yay) that made the tests
>> > run with multiple workers. This highlighted an issue with the Neutron
>> > unit testing infrastructure, which is fixed here:
>> > https://review.openstack.org/#/c/217379/
>> >
>> > With all three patches merged we should be good to go.
>>
>> Well, with all 3 of these we should be much better for sure. There are
>> probably additional issues causing intermittent failures which should be
>> looked at. These 3 are definitely masking anything else.
>>
>
> Sadly, since the issues are independent, it is very likely for one of the
> patch to fail jenkins tests for one of the other two issues.
> If the situation persists is it crazy to conside switching neutron-py34
> and neutron-functional to non-voting until these patches merge.
> Neutron cores might abstain from approving patches (unless trivial or
> documentation) while these jobs are non-voting.
>

We have two of the three merged. The Neutron functional tests are no longer
part of the gate queue, only the check queue,
and the Tempest router_reschedule test will no longer fail as part of the
DVR job. This means that the py34 patch now has
a better chance of going in.


>
>
>>
>> https://etherpad.openstack.org/p/gate-fire-2015-08-28 is a set of
>> patches to promote for things causing races in the gate (we've got a
>> cinder one was well). If other issues are known with fixes posted,
>> please feel free to add them with comments.
>>
>
>
>
>>
>>         -Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150828/70109598/attachment.html>


More information about the OpenStack-dev mailing list