Open Stack

On 1/10/2017 10:02 AM, Jordan Pittier wrote:
> Hi,
> I don't know if you've noticed but
> the gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial job has a
> high rate of false negative. I've queried Gerrit and analysed all the
> "Verified -2" messages left by Jenkins (i.e Gate failure) for the last
> 30 days. (script is here [1]).
>
> On project openstack/nova: For the last 58 times where
> gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND
> jenkins left a 'Verified -2' message, the job failed 48 times and
> succeeded 10 times.
>
> On project openstack/tempest: For the last 25 times where
> gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial ran AND
> jenkins left a 'Verified -2' message, the job failed 14 times and
> succeeded 11 times.
>
> In order words, when there's a gate failure
> gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial is the main
> culprit, by a significant margin.
>
> I am Tempest core reviewer and this bugs me because it slows the
> development of the project I care for reasons that I don't really
> understand. I am going to propose a change to make this job non voting
> on openstack/tempest.
>
> Jordan
>
> [1]
> : https://github.com/JordanP/openstack-snippets/blob/master/analyse-gate-failures/analyse_gate_failures.py
>
> <http://go.scality.com/acton/media/18585/gartner-magic-quadrant-object-storage?utm_campaign=MQ&utm_medium=Email&utm_source=signatures>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

The ceph job has had a high failure rate the last month or more. It's 
been very whack-a-mole from what I've seen when I'm digging into issues. 
There are still some open unresolved bugs being tracked against that job 
in the e-r status page:

http://status.openstack.org//elastic-recheck/index.html

We've fixed a few issues already (device not found on volume detach race 
in nova was one, and some cinder capacity filter issues in another), but 
what's out there now is still an issue and I'm not aware of a ton of 
focus on fixing those. jbernard probably knows the latest but unless 
there are good fixes just waiting for review, then I'm probably OK with 
making it non-voting too.

The most recent bug I reported against that job was due to the c-vol 
service completely dropping out in that job for some reason and then the 
cinder scheduler couldn't build any volumes. Very weird.

-- 

Thanks,

Matt Riedemann