<HTML>
<HEAD>
<TITLE>Re: [Openstack] Intermittent devstack-gate failures</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hmmm that makes it hard to develop on RHEL again (or I guess fedora 16?)<BR>
<BR>
Durn. Any reasoning behind the 0.9.7 change? Just out of curiosity...<BR>
<BR>
On 6/12/12 6:11 PM, "Dan Prince" <<a href="dprince@redhat.com">dprince@redhat.com</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hey Jim,<BR>
<BR>
I actually turned off SmokeStack earlier today for potentially the same reason. I was out for a couple days last week and haven't quite put my finger on all the things that are wrong. I'm seeing about half of the functional test runs fail.<BR>
<BR>
This issue seems to be the cause of most of my failures:<BR>
<BR>
<a href="https://bugs.launchpad.net/nova/+bug/1010291">https://bugs.launchpad.net/nova/+bug/1010291</a><BR>
<BR>
I'm also dealing with the fact that Nova now requires libvirt 0.9.7 or later (I think) due to some of the refactoring. This is fine but it does mean I can't fully test things on Fedora 16 like I used to (reboot is out for example).<BR>
<BR>
That is what I've seen.<BR>
<BR>
Dan<BR>
<BR>
----- Original Message -----<BR>
> From: "James E. Blair" <<a href="corvus@inaugust.com">corvus@inaugust.com</a>><BR>
> To: "OpenStack Mailing List" <<a href="openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>><BR>
> Sent: Tuesday, June 12, 2012 7:25:01 PM<BR>
> Subject: [Openstack] Intermittent devstack-gate failures<BR>
><BR>
> Hi,<BR>
><BR>
> It looks like there are intermittent, but frequent, failures in the<BR>
> devstack-gate. This suggests a non-deterministic bug has crept into<BR>
> some piece of OpenStack software.<BR>
><BR>
> In this kind of situation, certainly could keep re-approving changes<BR>
> in<BR>
> the hope that they will pass the test and merge, but it would be<BR>
> better<BR>
> to fix the underlying problem. Simply re-approving is mostly just<BR>
> going<BR>
> to make the queue longer.<BR>
><BR>
> Note that the output from Jenkins has changed recently. I've seen<BR>
> some<BR>
> people misconstrue some normal parts of the test process as errors.<BR>
> In<BR>
> particular, this message from Jenkins is not an error:<BR>
><BR>
> Looks like the node went offline during the build. Check the slave<BR>
> log<BR>
> for the details.<BR>
><BR>
> That's a normal part of the way the devstack-gate tests run, where we<BR>
> add a machine to Jenkins as a slave, run the tests, and remove it<BR>
> from<BR>
> the list of slaves before it's done. This is to accommodate the<BR>
> one-shot nature of devstack based testing. It doesn't interfere with<BR>
> the results.<BR>
><BR>
> To find out why a test failed, you should scroll up a bit to the<BR>
> devstack exercise output, which normally looks like this:<BR>
><BR>
> *********************************************************************<BR>
> SUCCESS: End DevStack Exercise: ./exercises/volumes.sh<BR>
> *********************************************************************<BR>
> =====================================================================<BR>
> SKIP boot_from_volume<BR>
> SKIP client-env<BR>
> SKIP quantum<BR>
> SKIP swift<BR>
> PASS aggregates<BR>
> PASS bundle<BR>
> PASS client-args<BR>
> PASS euca<BR>
> PASS floating_ips<BR>
> PASS sec_groups<BR>
> PASS volumes<BR>
> =====================================================================<BR>
><BR>
> Everything after that point is test running boilerplate. I'll add<BR>
> some<BR>
> echo statements to that effect in the future.<BR>
><BR>
> Finally, it may be a little difficult to pinpoint when this started.<BR>
> A<BR>
> number of devstack-gate tests have passed recently without actually<BR>
> running any tests, due to an issue with one of our OpenStack based<BR>
> node<BR>
> providers. We are eating our own dogfood.<BR>
><BR>
> -Jim<BR>
><BR>
> _______________________________________________<BR>
> Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><BR>
> Post to : <a href="openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><BR>
> Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><BR>
> More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><BR>
><BR>
<BR>
_______________________________________________<BR>
Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><BR>
Post to : <a href="openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><BR>
Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><BR>
More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>