Open Stack

On 24 August 2016 at 10:30, Armando M. <armamig at gmail.com> wrote:

>
>
> On 22 August 2016 at 18:25, Armando M. <armamig at gmail.com> wrote:
>
>>
>>
>> On 22 August 2016 at 09:51, Armando M. <armamig at gmail.com> wrote:
>>
>>> Neutrinos,
>>>
>>> Since the merge of test [1], a race has surfaced, which leads to failure
>>> as reported in [2]. A number of Neutron's jobs are affected, and even
>>> though the failure is only intermittent, it is particularly bad for the
>>> linux bridge job.
>>>
>>> If I can ask you to hold your +2/+W until [3] puts the fire out, zuul
>>> will be grateful.
>>>
>>
>> An update:
>>
>> We have [1] in the gate queue, and Kevin is going full steam with [2],
>> which will mitigate the other intermittent issues we're facing in
>> functional and unit tests. Please bear with us.
>>
>> Cheers,
>> Armando
>>
>> [1] https://review.openstack.org/#/c/358753/
>> [2] https://review.openstack.org/#/c/356530/
>>
>>
>>>
>>> Thanks,
>>> Armando
>>>
>>> [1] https://review.openstack.org/#/c/327191/
>>> [2] https://bugs.launchpad.net/neutron/+bug/1615710
>>> [3] https://review.openstack.org/#/c/358753/
>>>
>>
>
> One more update:
>
> We're not quite out of the woods, yet so approve/recheck gently please.
>
> Things have become slightly better but we still have a pending fix for
> Neutron [1]. Recently we also observed zuul nodes dying unexpectedly on the
> OSIC cloud (that runs Neutron on IPv6) and that had an impact on the
> overall sluggishness of the gate [2]. Once all of the fixes are in, we
> should be back in business...until the next fire!
>

Another (final) update:

Changes [1] have merged. That should take care of the infra node resets.

At this point, there's [2] to wrap up, but things look largely recovered
[3,4] (though we need [5] to fully see a current picture). Please keep an
eye on [6], and help report/address critical fixes in the busy times ahead.

Cheers,
Armando

[1]
https://review.openstack.org/#/q/Ia044fff2a1731ab6c04f82aea47096b425e0c0a0,n,z
[2] https://review.openstack.org/#/c/356530/
[3]
http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=3&fullscreen
[4]
http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=5&fullscreen
[5] https://review.openstack.org/#/c/358462/
[6]
https://bugs.launchpad.net/neutron/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.tag=gate-failure


>
> Cheers,
> Armando
>
> [1] https://review.openstack.org/#/c/356530/
> [2] https://bugs.launchpad.net/neutron/+bug/1616282
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160824/343325f9/attachment.html>