[openstack-dev] flaky tempest -- Top Offenders

Joe Gordon joe.gordon0 at gmail.com
Fri Sep 27 01:41:51 UTC 2013


Hi All,

As many of you may have suspected the gate has gotten less stable in the
past few days.  Turns out we have the numbers to prove it too!

http://graphite.openstack.org/graphlot/?width=586&from=00%3A00_20130919&_salt=1380244287.508&height=308&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2C%2224h%22)&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS%2C%2224h%22)&until=23%3A59_20130926&lineMode=staircase

So tempest started failing more right around the 24th, even though we are
in FeatureFreeze.

"FF ensures that sufficient share of the
ReleaseCycle<https://wiki.openstack.org/wiki/ReleaseCycle>
 is dedicated to QA, until we produce the first release candidates.
Limiting the changes that affect the behavior of the software allow for
consistent testing and efficient bugfixing."

https://wiki.openstack.org/wiki/FeatureFreeze

Thanks to the work we have been doing with logstash and elastic-recheck we
have very good numbers on the top offenders and when they began, the good
news is there are two bugs which we are hitting the most, so the top
offenders list has just two bugs. But there are still other unknown bugs
and lower priority ones out there too!


https://bugs.launchpad.net/tempest/+bug/1226337 -- Launchpad bug 1226337 in
tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern
flake failure" [High,Triaged]

Started on 9-23 with  408 hits! in the last 24 hours alone

http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZTpcIk5vdmFFeGNlcHRpb246IGlTQ1NJIGRldmljZSBub3QgZm91bmQgYXRcIiBBTkQgQGZpZWxkcy5idWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDI0NDY2ODQ5Nn0=


https://bugs.launchpad.net/tempest/+bug/1230407  -- Launchpad bug 1230407
in neutron "State change timeout exceeded" [Undecided,Confirmed]

Started on 9-25 with 66 hits in the last 24 hours alone

http://logstash.openstack.org/#eyJzZWFyY2giOiIgQG1lc3NhZ2U6XCJBc3NlcnRpb25FcnJvcjogU3RhdGUgY2hhbmdlIHRpbWVvdXQgZXhjZWVkZWQhXCIgQU5EIEBmaWVsZHMuYnVpbGRfc3RhdHVzOlwiRkFJTFVSRVwiIEFORCBAZmllbGRzLmZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODAyNDQ0MzM2NzZ9


Hopefully we can get both of these fixed very soon, so we can stabilize
gate again.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130926/7ada601f/attachment.html>


More information about the OpenStack-dev mailing list