<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">There is just one queue that is building up: notifications.info - there are no consumers for it.<div><br></div><div>I have upped the ulimit for the rabbitmq server (by changing /etc/default/rabbitmq-server) but that hasn’t made any difference.</div><div><br></div><div>Are you also upping the limits on the compute nodes? <br><div><br class="webkit-block-placeholder"></div><div>cheers</div><div>jc</div><div><br></div><div apple-content-edited="true">
-- <br>SWITCH<br>Jens-Christian Fischer, Peta Solutions<br>Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>phone +41 44 268 15 15, direct +41 44 268 15 71<br><a href="mailto:jens-christian.fischer@switch.ch">jens-christian.fischer@switch.ch</a><br>http://www.switch.ch<br><br>http://www.switch.ch/stories
</div>
<br><div><div>On 11.07.2014, at 12:50, Brebner, Gavin <<a href="mailto:gavin.brebner@hp.com">gavin.brebner@hp.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-GB" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="WordSection1" style="page: WordSection1;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Delete issues in my experience are typically linked to messaging getting stuck (or dead hosts). I would have a look at the<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">RMQ queue lengths and see if any are building up.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Have you adjusted your open file descriptor limits ? The default Linux limit (1024) is NOT enough in our experience for any reasonable<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">sized system. We’ve seen that cause messaging outages.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> Gavin<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="border-style: none none none solid; border-left-color: blue; border-left-width: 1.5pt; padding: 0in 0in 0in 4pt;"><div><div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"><span class="Apple-converted-space"> </span>Jens-Christian Fischer [<a href="mailto:jens-christian.fischer@switch.ch">mailto:jens-christian.fischer@switch.ch</a>]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, July 11, 2014 12:32 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Thomas Bernard<br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Openstack] Unable to delete instances<o:p></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Did you ever come to the bottom of this?<o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">We are seeing the same thing with a brand new Icehouse / Ceph installation (using the stock distributions)<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Restarting nova-compute on the compute nodes usually makes the instances go away (or resetting the state, and then restarting nova-compute)<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">We see this problem intermittent but frequent enough to be really annoying.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">The log files have so far not shown anything conclusive: I got an “Unknown auth strategy” error, but according to <a href="https://bugs.launchpad.net/python-neutronclient/+bug/1297309" style="color: purple; text-decoration: underline;">https://bugs.launchpad.net/python-neutronclient/+bug/1297309</a> this is a red herring.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Anyone else running into this problem or having solved it?<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">cheers<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Jens-Christian<o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">-- <br>SWITCH<br>Jens-Christian Fischer, Peta Solutions<br>Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>phone +41 44 268 15 15, direct +41 44 268 15 71<br><a href="mailto:jens-christian.fischer@switch.ch" style="color: purple; text-decoration: underline;">jens-christian.fischer@switch.ch</a><br><a href="http://www.switch.ch/" style="color: purple; text-decoration: underline;">http://www.switch.ch</a><br><br><a href="http://www.switch.ch/stories" style="color: purple; text-decoration: underline;">http://www.switch.ch/stories</a><o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div><div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">On 22.05.2014, at 16:19, Thomas Bernard <<a href="mailto:tbe@atolcd.com" style="color: purple; text-decoration: underline;">tbe@atolcd.com</a>> wrote:<o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><br><br><o:p></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Hi,<br><br>I just finished my upgrade to icehouse, everything works just fine but... i can't delete my instances!<br><br>When i delete instance (from cli or from horizon) :<span class="Apple-converted-space"> </span><br><br>* The VM goes to "deleting"state<br>* Vm are destroy from KVM<br>* But nova list & horizon continue to display the "deleting" state<br><br>Some time after, the Vm state goes to "ERROR" and in nova-compute.log i can see : "Unauthorized: Unknown auth strategy\n" (from neutronclient)<br><br>Any idea?<br><br>Thx!<br><br><br><br><br><br>_______________________________________________<br>Mailing list:<span class="Apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" style="color: purple; text-decoration: underline;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>Post to :<span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org" style="color: purple; text-decoration: underline;">openstack@lists.openstack.org</a><br>Unsubscribe :<span class="Apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" style="color: purple; text-decoration: underline;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></div></blockquote></div></div></div></div></div></blockquote></div><br></div></body></html>