<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">we change /etc/security/limits.conf (actually add an entry to /etc/security/limits.conf.d) for the rabbit user.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">We just do that for the non-compute nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"> Gavin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Jens-Christian Fischer [mailto:jens-christian.fischer@switch.ch]
<br>
<b>Sent:</b> Friday, July 11, 2014 3:41 PM<br>
<b>To:</b> Brebner, Gavin<br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] Unable to delete instances<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is just one queue that is building up: notifications.info - there are no consumers for it.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have upped the ulimit for the rabbitmq server (by changing /etc/default/rabbitmq-server) but that hasn’t made any difference.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Are you also upping the limits on the compute nodes? <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">cheers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">jc<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- <br>
SWITCH<br>
Jens-Christian Fischer, Peta Solutions<br>
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>
phone +41 44 268 15 15, direct +41 44 268 15 71<br>
<a href="mailto:jens-christian.fischer@switch.ch">jens-christian.fischer@switch.ch</a><br>
<a href="http://www.switch.ch">http://www.switch.ch</a><br>
<br>
<a href="http://www.switch.ch/stories">http://www.switch.ch/stories</a> <o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On 11.07.2014, at 12:50, Brebner, Gavin <<a href="mailto:gavin.brebner@hp.com">gavin.brebner@hp.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Delete issues in my experience are typically linked to messaging getting stuck (or dead hosts). I would have a look at the</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">RMQ queue lengths and see if any are building up.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Have you adjusted your open file descriptor limits ? The default Linux limit (1024) is NOT enough in our experience for any reasonable</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sized system. We’ve seen that cause messaging outages.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Gavin</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Jens-Christian
Fischer [<a href="mailto:jens-christian.fischer@switch.ch">mailto:jens-christian.fischer@switch.ch</a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Friday, July 11, 2014 12:32 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Thomas Bernard<br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [Openstack] Unable to delete instances</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Did you ever come to the bottom of this?<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">We are seeing the same thing with a brand new Icehouse / Ceph installation (using the stock distributions)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Restarting nova-compute on the compute nodes usually makes the instances go away (or resetting the state, and then restarting nova-compute)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">We see this problem intermittent but frequent enough to be really annoying.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">The log files have so far not shown anything conclusive: I got an “Unknown auth strategy” error, but according to <a href="https://bugs.launchpad.net/python-neutronclient/+bug/1297309"><span style="color:purple">https://bugs.launchpad.net/python-neutronclient/+bug/1297309</span></a> this
is a red herring.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Anyone else running into this problem or having solved it?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">cheers<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Jens-Christian<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">-- <br>
SWITCH<br>
Jens-Christian Fischer, Peta Solutions<br>
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland<br>
phone +41 44 268 15 15, direct +41 44 268 15 71<br>
<a href="mailto:jens-christian.fischer@switch.ch"><span style="color:purple">jens-christian.fischer@switch.ch</span></a><br>
<a href="http://www.switch.ch/"><span style="color:purple">http://www.switch.ch</span></a><br>
<br>
<a href="http://www.switch.ch/stories"><span style="color:purple">http://www.switch.ch/stories</span></a><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On 22.05.2014, at 16:19, Thomas Bernard <<a href="mailto:tbe@atolcd.com"><span style="color:purple">tbe@atolcd.com</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi,<br>
<br>
I just finished my upgrade to icehouse, everything works just fine but... i can't delete my instances!<br>
<br>
When i delete instance (from cli or from horizon) :<span class="apple-converted-space"> </span><br>
<br>
* The VM goes to "deleting"state<br>
* Vm are destroy from KVM<br>
* But nova list & horizon continue to display the "deleting" state<br>
<br>
Some time after, the Vm state goes to "ERROR" and in nova-compute.log i can see : "Unauthorized: Unknown auth strategy\n" (from neutronclient)<br>
<br>
Any idea?<br>
<br>
Thx!<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list:<span class="apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="color:purple">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><br>
Post to :<span class="apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org"><span style="color:purple">openstack@lists.openstack.org</span></a><br>
Unsubscribe :<span class="apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><span style="color:purple">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>