[openstack-dev] [neutron] Race conditions in fwaas that impact the gate

Salvatore Orlando salv.orlando at gmail.com
Tue Aug 11 23:32:34 UTC 2015


I have been hit by these failures as well.
I think you did well by bumping out that revert from the queue; I think it
simply cures the sympton possibly affecting correct operations of the
firewall service.
If we are looking at removing the sympton on the API job, than I'd skip the
failing tests while somebody figures out what's going on (unless the team
decides that it is better to revert again multiple workers).

However, I think the issue might not be limited at firewall. I've seen a
worrying spike in rally failures [1]. Since it's non-voting probably
developers do not care a lot about it, but it provides very useful
insights. I am looking at rally logs now - at the moment I have not yet a
clear idea of the root cause of such failures.

Salvatore

[1]
http://graphite.openstack.org/render/?width=840&height=308&_salt=1439335659.449&target=hitcount%28stats.zuul.pipeline.check.job.gate-rally-dsvm-neutron-neutron.FAILURE%2C%221h%22%29&from=-72hours


On 12 August 2015 at 00:21, Sean M. Collins <sean at coreitpro.com> wrote:

> Hello,
>
> Today has been an exciting day, to say the least. Earlier today I was
> pinged on IRC about some firewall as a service unit test failures that
> were blocking patches from being merged, such as
> https://review.openstack.org/#/c/211537/.
>
> Neutron devs started poking around a bit and discussing on the IRC channel.
>
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-08-11.log.html#t2015-08-11T16:59:13
>
> I've started to dig a little bit and document what I've found on this
> bug.
>
> https://bugs.launchpad.net/neutron/+bug/1483875
>
> There was a change recently merged in devstack-gate which changes the
> MySQL database driver and the number of workers -
> https://review.openstack.org/#/c/210649/
> which might be what is triggering the race condition - but I'm honestly
> not sure.
>
> I proposed a revert to a section of the FwaaS code, but frankly I'm not
> sure if this will fix the problem - https://review.openstack.org/211677
> - so I bumped it out of the merge queue when my anxiety reached maximum.
> I'm just not confident enough about my knowledge of the FwaaS codebase
> to really be making these kinds of changes.
>
> Is there anyone that has any insights?
>
>
> --
> Sean M. Collins
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150812/cf440748/attachment.html>


More information about the OpenStack-dev mailing list