<div dir="ltr">Hi Michael, Abel,<div>thank you for the link and suggestion.</div><div><br></div><div>Belmiro</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 21, 2014 at 7:58 PM, Tim Bell <span dir="ltr"><<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Following
<a href="https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes" target="_blank">https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes</a>, I’ve tagged
</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><a href="https://bugs.launchpad.net/oslo/+bug/856764" target="_blank">https://bugs.launchpad.net/oslo/+bug/856764</a> with
</span><b><span style="font-size:10.5pt;font-family:"Arial Unicode MS","sans-serif";color:#333333;background:white">havana-backport-potential</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Tim Bell
<br>
<b>Sent:</b> 21 May 2014 17:39<br>
<b>To:</b> 'Michael Dorman'; Belmiro Moreira; OpenStack Operators<br>
<b>Subject:</b> RE: [Openstack-operators] nova-compute not consuming messages<u></u><u></u></span></p>
</div>
</div><div class="">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Looks like a good stable backport candidate<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Tim<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</div><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Michael Dorman [<a href="mailto:mdorman@godaddy.com" target="_blank">mailto:mdorman@godaddy.com</a>]
<br></span></p><div class="">
<b>Sent:</b> 21 May 2014 17:22<br>
<b>To:</b> Belmiro Moreira; OpenStack Operators<br>
</div><div class=""><b>Subject:</b> Re: [Openstack-operators] nova-compute not consuming messages<u></u><u></u></div><p></p>
</div>
</div><div class="">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Actually, I lied. This change has been merged and is part of Icehouse.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">We are still running on Havana, though, and haven’t actually tried it. People we talked to have had success with it, though.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Michael Dorman <<a href="mailto:mdorman@godaddy.com" target="_blank">mdorman@godaddy.com</a>><br>
<b>Date: </b>Wednesday, May 21, 2014 at 9:14 AM<br>
<b>To: </b>Belmiro Moreira <<a href="mailto:moreira.belmiro.email.lists@gmail.com" target="_blank">moreira.belmiro.email.lists@gmail.com</a>>, OpenStack Operators <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [Openstack-operators] nova-compute not consuming messages<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">We see the same behavior, and a lot of other people have this problem, too. It’s related to how the Rabbit client code is handling disconnects/reconnects. Do
you happen to be connecting to Rabbit through a load balancer?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">We don’t have a create solution for this, but talked to several folks at the Summit for ideas.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">See this bug: <a href="https://bugs.launchpad.net/oslo/+bug/856764" target="_blank">https://bugs.launchpad.net/oslo/+bug/856764</a> and this review: <a href="https://review.openstack.org/#/c/76686/" target="_blank">https://review.openstack.org/#/c/76686/</a> for
a possible solution. This has not been merged in yet, though.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Mike<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Belmiro Moreira <<a href="mailto:moreira.belmiro.email.lists@gmail.com" target="_blank">moreira.belmiro.email.lists@gmail.com</a>><br>
<b>Date: </b>Wednesday, May 21, 2014 at 8:05 AM<br>
<b>To: </b>OpenStack Operators <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>><br>
<b>Subject: </b>[Openstack-operators] nova-compute not consuming messages<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><u></u> <u></u></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">In our infrastructure we are observing that some compute nodes are reporting the state correctly but not consuming messages.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">For the scheduler they are available, so it schedules VMs on them, and then the messages are pilling up in the compute.”host” queues.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Restarting nova-compute solves the problem. All old messages in the queue are consumed and they start behaving properly until next time…<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">I don’t get anything interesting from the logs.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">We are using rabbit 3.2.4 clustered with mirror queues and Havana 2.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Anyone observed the same symptoms?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Belmiro<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div></div>
</div>
</div>
</div>
</blockquote></div><br></div>