<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">It seems that I as chasing down the wrong rabbit hole…<div><br></div><div>Our controller node runs on a VM and has had sever hiccups the last few days (pausing for up to 10 seconds from time to time). We have moved the VM to another host, the pauses disappeared, and I can delete as many VMs at a time as I like…</div><div><br></div><div>cheers</div><div>jc</div><div><br><div apple-content-edited="true">
-- <br>SWITCH<br>Jens-Christian Fischer, Peta Solutions<br>Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>phone +41 44 268 15 15, direct +41 44 268 15 71<br><a href="mailto:jens-christian.fischer@switch.ch">jens-christian.fischer@switch.ch</a><br>http://www.switch.ch<br><br>http://www.switch.ch/stories
</div>
<br><div><div>On 11.07.2014, at 15:45, Brebner, Gavin <<a href="mailto:gavin.brebner@hp.com">gavin.brebner@hp.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-GB" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="WordSection1" style="page: WordSection1;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">we change /etc/security/limits.conf (actually add an entry to /etc/security/limits.conf.d) for the rabbit user.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">We just do that for the non-compute nodes.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> Gavin<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="border-style: none none none solid; border-left-color: blue; border-left-width: 1.5pt; padding: 0in 0in 0in 4pt;"><div><div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"><span class="Apple-converted-space"> </span>Jens-Christian Fischer [<a href="mailto:jens-christian.fischer@switch.ch">mailto:jens-christian.fischer@switch.ch</a>]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, July 11, 2014 3:41 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Brebner, Gavin<br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Openstack] Unable to delete instances<o:p></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">There is just one queue that is building up: notifications.info - there are no consumers for it.<o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">I have upped the ulimit for the rabbitmq server (by changing /etc/default/rabbitmq-server) but that hasn’t made any difference.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Are you also upping the limits on the compute nodes? <o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">cheers<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">jc<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">-- <br>SWITCH<br>Jens-Christian Fischer, Peta Solutions<br>Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>phone +41 44 268 15 15, direct +41 44 268 15 71<br><a href="mailto:jens-christian.fischer@switch.ch" style="color: purple; text-decoration: underline;">jens-christian.fischer@switch.ch</a><br><a href="http://www.switch.ch/" style="color: purple; text-decoration: underline;">http://www.switch.ch</a><br><br><a href="http://www.switch.ch/stories" style="color: purple; text-decoration: underline;">http://www.switch.ch/stories</a><o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><o:p> </o:p></div><div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">On 11.07.2014, at 12:50, Brebner, Gavin <<a href="mailto:gavin.brebner@hp.com" style="color: purple; text-decoration: underline;">gavin.brebner@hp.com</a>> wrote:<o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><br><br><o:p></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Delete issues in my experience are typically linked to messaging getting stuck (or dead hosts). I would have a look at the</span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">RMQ queue lengths and see if any are building up.</span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Have you adjusted your open file descriptor limits ? The default Linux limit (1024) is NOT enough in our experience for any reasonable</span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">sized system. We’ve seen that cause messaging outages.</span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> Gavin</span><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span><o:p></o:p></div></div><div style="border-style: none none none solid; border-left-color: blue; border-left-width: 1.5pt; padding: 0in 0in 0in 4pt;"><div><div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span></span><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">Jens-Christian Fischer [<a href="mailto:jens-christian.fischer@switch.ch" style="color: purple; text-decoration: underline;">mailto:jens-christian.fischer@switch.ch</a>]<span class="apple-converted-space"> </span><br><b>Sent:</b><span class="apple-converted-space"> </span>Friday, July 11, 2014 12:32 PM<br><b>To:</b><span class="apple-converted-space"> </span>Thomas Bernard<br><b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org" style="color: purple; text-decoration: underline;">openstack@lists.openstack.org</a><br><b>Subject:</b><span class="apple-converted-space"> </span>Re: [Openstack] Unable to delete instances</span><o:p></o:p></div></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Did you ever come to the bottom of this?<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">We are seeing the same thing with a brand new Icehouse / Ceph installation (using the stock distributions)<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Restarting nova-compute on the compute nodes usually makes the instances go away (or resetting the state, and then restarting nova-compute)<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">We see this problem intermittent but frequent enough to be really annoying.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">The log files have so far not shown anything conclusive: I got an “Unknown auth strategy” error, but according to <a href="https://bugs.launchpad.net/python-neutronclient/+bug/1297309" style="color: purple; text-decoration: underline;"><span style="color: purple;">https://bugs.launchpad.net/python-neutronclient/+bug/1297309</span></a> this is a red herring.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Anyone else running into this problem or having solved it?<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">cheers<o:p></o:p></div></div><div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Jens-Christian<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">-- <br>SWITCH<br>Jens-Christian Fischer, Peta Solutions<br>Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>phone +41 44 268 15 15, direct +41 44 268 15 71<br><a href="mailto:jens-christian.fischer@switch.ch" style="color: purple; text-decoration: underline;"><span style="color: purple;">jens-christian.fischer@switch.ch</span></a><br><a href="http://www.switch.ch/" style="color: purple; text-decoration: underline;"><span style="color: purple;">http://www.switch.ch</span></a><br><br><a href="http://www.switch.ch/stories" style="color: purple; text-decoration: underline;"><span style="color: purple;">http://www.switch.ch/stories</span></a><o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"> <o:p></o:p></div></div><div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">On 22.05.2014, at 16:19, Thomas Bernard <<a href="mailto:tbe@atolcd.com" style="color: purple; text-decoration: underline;"><span style="color: purple;">tbe@atolcd.com</span></a>> wrote:<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><br><br><br><o:p></o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">Hi,<br><br>I just finished my upgrade to icehouse, everything works just fine but... i can't delete my instances!<br><br>When i delete instance (from cli or from horizon) :<span class="apple-converted-space"> </span><br><br>* The VM goes to "deleting"state<br>* Vm are destroy from KVM<br>* But nova list & horizon continue to display the "deleting" state<br><br>Some time after, the Vm state goes to "ERROR" and in nova-compute.log i can see : "Unauthorized: Unknown auth strategy\n" (from neutronclient)<br><br>Any idea?<br><br>Thx!<br><br><br><br><br><br>_______________________________________________<br>Mailing list:<span class="apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" style="color: purple; text-decoration: underline;"><span style="color: purple;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><br>Post to :<span class="apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org" style="color: purple; text-decoration: underline;"><span style="color: purple;">openstack@lists.openstack.org</span></a><br>Unsubscribe :<span class="apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" style="color: purple; text-decoration: underline;"><span style="color: purple;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br></div></body></html>