Hi All, As many of you may have suspected the gate has gotten less stable in the past few days. Turns out we have the numbers to prove it too! http://graphite.openstack.org/graphlot/?width=586&from=00%3A00_20130919&_salt=1380244287.508&height=308&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2C%2224h%22)&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS%2C%2224h%22)&until=23%3A59_20130926&lineMode=staircase So tempest started failing more right around the 24th, even though we are in FeatureFreeze. "FF ensures that sufficient share of the ReleaseCycle<https://wiki.openstack.org/wiki/ReleaseCycle> is dedicated to QA, until we produce the first release candidates. Limiting the changes that affect the behavior of the software allow for consistent testing and efficient bugfixing." https://wiki.openstack.org/wiki/FeatureFreeze Thanks to the work we have been doing with logstash and elastic-recheck we have very good numbers on the top offenders and when they began, the good news is there are two bugs which we are hitting the most, so the top offenders list has just two bugs. But there are still other unknown bugs and lower priority ones out there too! https://bugs.launchpad.net/tempest/+bug/1226337 -- Launchpad bug 1226337 in tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [High,Triaged] Started on 9-23 with 408 hits! in the last 24 hours alone http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZTpcIk5vdmFFeGNlcHRpb246IGlTQ1NJIGRldmljZSBub3QgZm91bmQgYXRcIiBBTkQgQGZpZWxkcy5idWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDI0NDY2ODQ5Nn0= https://bugs.launchpad.net/tempest/+bug/1230407 -- Launchpad bug 1230407 in neutron "State change timeout exceeded" [Undecided,Confirmed] Started on 9-25 with 66 hits in the last 24 hours alone http://logstash.openstack.org/#eyJzZWFyY2giOiIgQG1lc3NhZ2U6XCJBc3NlcnRpb25FcnJvcjogU3RhdGUgY2hhbmdlIHRpbWVvdXQgZXhjZWVkZWQhXCIgQU5EIEBmaWVsZHMuYnVpbGRfc3RhdHVzOlwiRkFJTFVSRVwiIEFORCBAZmllbGRzLmZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODAyNDQ0MzM2NzZ9 Hopefully we can get both of these fixed very soon, so we can stabilize gate again. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130926/7ada601f/attachment.html>