<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 26, 2013 at 6:41 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div>As many of you may have suspected the gate has gotten less stable in the past few days.  Turns out we have the numbers to prove it too!</div>

<div><br></div><div><div><a href="http://graphite.openstack.org/graphlot/?width=586&from=00%3A00_20130919&_salt=1380244287.508&height=308&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2C%2224h%22)&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS%2C%2224h%22)&until=23%3A59_20130926&lineMode=staircase" target="_blank">http://graphite.openstack.org/graphlot/?width=586&from=00%3A00_20130919&_salt=1380244287.508&height=308&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2C%2224h%22)&target=summarize(stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS%2C%2224h%22)&until=23%3A59_20130926&lineMode=staircase</a></div>


</div><div><br></div><div>So tempest started failing more right around the 24th, even though we are in FeatureFreeze.</div><div><br></div><div>"<span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px">FF ensures that sufficient share of the</span><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px"> </span><a href="https://wiki.openstack.org/wiki/ReleaseCycle" title="ReleaseCycle" style="font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px;color:rgb(0,136,204);text-decoration:none" target="_blank">ReleaseCycle</a><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px"> </span><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px">is dedicated to QA, until we produce the first release candidates. Limiting the changes that affect the behavior of the software allow for consistent testing and efficient bugfixing.</span><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px">"</span></div>


<div><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px"><br></span></div><div><a href="https://wiki.openstack.org/wiki/FeatureFreeze" target="_blank">https://wiki.openstack.org/wiki/FeatureFreeze</a><span style="color:rgb(51,51,51);font-family:'Arial Unicode MS',Arial,sans-serif;font-size:14px;line-height:20px"><br>


</span></div><div><br></div><div>Thanks to the work we have been doing with logstash and elastic-recheck we have very good numbers on the top offenders and when they began, the good news is there are two bugs which we are hitting the most, so the top offenders list has just two bugs. But there are still other unknown bugs and lower priority ones out there too!</div>


<div><br></div><div><br></div><div><div><a href="https://bugs.launchpad.net/tempest/+bug/1226337" target="_blank">https://bugs.launchpad.net/tempest/+bug/1226337</a> -- Launchpad bug 1226337 in tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [High,Triaged]</div>


<div><br></div><div>Started on 9-23 with  408 hits! in the last 24 hours alone</div></div></div></blockquote><div><br></div><div>That should have read 130 hits in the last 24 hours and over 408 in last 7 days. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir="ltr"><div><div><br></div><div><a href="http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZTpcIk5vdmFFeGNlcHRpb246IGlTQ1NJIGRldmljZSBub3QgZm91bmQgYXRcIiBBTkQgQGZpZWxkcy5idWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDI0NDY2ODQ5Nn0=" target="_blank">http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZTpcIk5vdmFFeGNlcHRpb246IGlTQ1NJIGRldmljZSBub3QgZm91bmQgYXRcIiBBTkQgQGZpZWxkcy5idWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDI0NDY2ODQ5Nn0=</a></div>


</div><div><br></div><div><br></div><div><a href="https://bugs.launchpad.net/tempest/+bug/1230407" target="_blank">https://bugs.launchpad.net/tempest/+bug/1230407</a>  -- Launchpad bug 1230407 in neutron "State change timeout exceeded" [Undecided,Confirmed]</div>


<div><br></div><div>Started on 9-25 with 66 hits in the last 24 hours alone</div><div><br></div><div><a href="http://logstash.openstack.org/#eyJzZWFyY2giOiIgQG1lc3NhZ2U6XCJBc3NlcnRpb25FcnJvcjogU3RhdGUgY2hhbmdlIHRpbWVvdXQgZXhjZWVkZWQhXCIgQU5EIEBmaWVsZHMuYnVpbGRfc3RhdHVzOlwiRkFJTFVSRVwiIEFORCBAZmllbGRzLmZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODAyNDQ0MzM2NzZ9" target="_blank">http://logstash.openstack.org/#eyJzZWFyY2giOiIgQG1lc3NhZ2U6XCJBc3NlcnRpb25FcnJvcjogU3RhdGUgY2hhbmdlIHRpbWVvdXQgZXhjZWVkZWQhXCIgQU5EIEBmaWVsZHMuYnVpbGRfc3RhdHVzOlwiRkFJTFVSRVwiIEFORCBAZmllbGRzLmZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODAyNDQ0MzM2NzZ9</a><br>


</div><div><br></div><div><br></div><div>Hopefully we can get both of these fixed very soon, so we can stabilize gate again.</div><div><br></div></div>
</blockquote></div><br></div></div>