<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">We’ve had a lot of issues with Icehouse related to rabbitMQ. Basically the change from openstack.rpc to oslo.messaging broke things. These things are now fixed in oslo.messaging version 1.5.1, there is still an issue with heartbeats and that patch is making it’s way through review process now.<div class=""><br class=""></div><div class=""><a href="https://review.openstack.org/#/c/146047/" class="">https://review.openstack.org/#/c/146047/</a></div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Sam</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On 16 Jan 2015, at 10:55 am, sridhar basam <<a href="mailto:sridhar.basam@gmail.com" class="">sridhar.basam@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><p dir="ltr" class=""><br class="">
If you are using ha queues, use a version of rabbitmq > 3.3.0. There was a change in that version where consumption on queues was automatically enabled when a master election for a queue happened. Previous versions only informed clients that they had to reconsume on a queue. It was the clients responsibility to start consumption on a queue.</p><p dir="ltr" class=""> Make sure you enable tcp keepalives to a low enough value in case you have a firewall device in between your rabbit server and it's consumers.</p><p dir="ltr" class="">Monitor consumers on your rabbit infrastructure using 'rabbitmqctl list_queues name messages consumers'. Consumers on fanout queues is going to depend on the number of services of any type you have in your environment.</p><p dir="ltr" class=""> Sri<br class="">
</p>
<div class="gmail_quote">On Jan 15, 2015 6:27 PM, "Michael Dorman" <<a href="mailto:mdorman@godaddy.com" class="">mdorman@godaddy.com</a>> wrote:<br type="attribution" class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap: break-word; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">
<div class="">Here is the bug I’ve been tracking related to this for a while. I haven’t really kept up to speed with it, so I don’t know the current status.</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://bugs.launchpad.net/nova/+bug/856764" target="_blank" class="">https://bugs.launchpad.net/nova/+bug/856764</a></div>
<div class=""><br class="">
</div>
<div class="">
<div class=""></div>
</div>
</div>
</div>
<div class=""><br class="">
</div>
<span class="">
<div style="font-family: Calibri; font-size: 12pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">From: </span>Kris Lindgren <<a href="mailto:klindgren@godaddy.com" target="_blank" class="">klindgren@godaddy.com</a>><br class="">
<span style="font-weight:bold" class="">Date: </span>Thursday, January 15, 2015 at 12:10 PM<br class="">
<span style="font-weight:bold" class="">To: </span>Gustavo Randich <<a href="mailto:gustavo.randich@gmail.com" target="_blank" class="">gustavo.randich@gmail.com</a>>, OpenStack Operators <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank" class="">openstack-operators@lists.openstack.org</a>><br class="">
<span style="font-weight:bold" class="">Subject: </span>Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="word-wrap: break-word; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">During the Atlanta ops meeting this topic came up and I specifically mentioned about adding a "no-op" or healthcheck ping to the rabbitmq stuff to both nova & neutron. The dev's in the room looked at me like I was crazy, but it was so that we could exactly
catch issues as you described. I am also interested if any one knows of a lightweight call that could be used to verify/confirm rabbitmq connectivity as well. I haven't been able to devote time to dig into it. Mainly because if one client is having issues
- you will notice other clients are having similar/silent errors and a restart of all the things is the easiest way to fix, for us atleast.</div>
<div class="">
<div class="">
<div class="">____________________________________________</div>
<div class=""> </div>
<div class="">Kris Lindgren</div>
<div class="">Senior Linux Systems Engineer</div>
<div class="">GoDaddy, LLC.</div>
</div>
<div class=""><br class="">
</div>
</div>
</div>
<div class=""><br class="">
</div>
<span class="">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">From: </span>Gustavo Randich <<a href="mailto:gustavo.randich@gmail.com" target="_blank" class="">gustavo.randich@gmail.com</a>><br class="">
<span style="font-weight:bold" class="">Date: </span>Thursday, January 15, 2015 at 11:53 AM<br class="">
<span style="font-weight:bold" class="">To: </span>"<a href="mailto:openstack-operators@lists.openstack.org" target="_blank" class="">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank" class="">openstack-operators@lists.openstack.org</a>><br class="">
<span style="font-weight:bold" class="">Subject: </span>Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class="">Just to add one more background scenario, we also had similar problems trying to load balance rabbitmq via F5 Big IP LTM. For that reason we don't use it now. Our installation is a single rabbitmq instance and no intermediaries (albeit network switches).
We use Folsom and Icehouse, the problem being perceived more in Icehouse nodes.<br class="">
<br class="">
</div>
We are already monitoring message queue size, but we would like to pinpoint in semi-realtime the specific hosts/racks/network paths experiencing the "stale connection" before a user complains about an operation being stuck, or even hosts with no such pending
operations but already "disconnected" -- we also could diagnose possible network causes and avoid massive service restarting.<br class="">
<br class="">
</div>
<div class="">So, for now, if someone knows about a cheap and quick openstack operation that triggers a message interchange between rabbitmq and nova-compute and a way of checking the result it would be great.<br class="">
<br class="">
<br class="">
</div>
<br class="">
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Thu, Jan 15, 2015 at 1:45 PM, Kris G. Lindgren <span dir="ltr" class="">
<<a href="mailto:klindgren@godaddy.com" target="_blank" class="">klindgren@godaddy.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap: break-word; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">We did have an issue using celery on an internal application that we wrote - but I believe it was fixed after much failover testing and code changes. We also use logstash via rabbitmq and haven't noticed any issues there either.</div>
<div class=""><br class="">
</div>
<div class="">So this seems to be just openstack/oslo related.</div>
<div class=""><br class="">
</div>
<div class="">We have tried a number of different configurations - all of them had their issues. We started out listing all the members in the cluster on the rabbit_hosts line. This worked most of the time without issue, until we would restart one of the servers,
then it seemed like the clients wouldn't figure out they were disconnected and reconnect to the next host. </div>
<div class=""><br class="">
</div>
<div class="">In an attempt to solve that we moved to using harpoxy to present a vip that we configured in the rabbit_hosts line. This created issues with long lived connections disconnects and a bunch of other issues. In our production environment we moved to load
balanced rabbitmq, but using a real loadbalancer, and don’t have the weird disconnect issues. However, anytime we reboot/take down a rabbitmq host or pull a member from the cluster we have issues, or if their is a network disruption we also have issues.</div>
<div class=""><br class="">
</div>
<div class="">Thinking the best course of action is to move rabbitmq off on to its own box and to leave it alone.</div>
<div class=""><br class="">
</div>
<div class="">Does anyone have a rabbitmq setup that works well and doesn’t have random issues when pulling nodes for maintenance?</div>
<span class="">
<div class="">
<div class="">
<div class="">____________________________________________</div>
<div class=""> </div>
<div class="">Kris Lindgren</div>
<div class="">Senior Linux Systems Engineer</div>
<div class="">GoDaddy, LLC.</div>
</div>
<div class=""><br class="">
</div>
</div>
</span></div>
<div class=""><br class="">
</div>
<span class="">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">From: </span>Joe Topjian <<a href="mailto:joe@topjian.net" target="_blank" class="">joe@topjian.net</a>><br class="">
<span style="font-weight:bold" class="">Date: </span>Thursday, January 15, 2015 at 9:29 AM<br class="">
<span style="font-weight:bold" class="">To: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com" target="_blank" class="">klindgren@godaddy.com</a>><br class="">
<span style="font-weight:bold" class="">Cc: </span>"<a href="mailto:openstack-operators@lists.openstack.org" target="_blank" class="">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank" class="">openstack-operators@lists.openstack.org</a>><br class="">
<span style="font-weight:bold" class="">Subject: </span>Re: [Openstack-operators] Way to check compute <-> rabbitmq connectivity<br class="">
</div>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div dir="ltr" class="">Hi Kris,
<div class=""><br class="">
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap: break-word; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class=""> Our experience is pretty much the same on anything that is using rabbitmq - not just nova-compute. </div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Just to clarify: have you experienced this outside of OpenStack (or Oslo)? </div>
<div class=""><br class="">
</div>
<div class="">We've seen similar issues with rabbitmq and OpenStack. We used to run rabbit through haproxy and tried a myriad of options like setting no timeouts, very very long timeouts, etc, but would always eventually see similar issues as described.</div>
<div class=""><br class="">
</div>
<div class="">Last month, we reconfigured all OpenStack components to use the `rabbit_hosts` option with all nodes in our cluster listed. So far this has worked well, though I probably just jinxed myself. :)</div>
<div class=""><br class="">
</div>
<div class="">We still have other services (like Sensu) using the same rabbitmq cluster and accessing it through haproxy. We've never had any issues there.</div>
<div class=""><br class="">
</div>
<div class="">What's also strange is that I have another OpenStack deployment (from Folsom to Icehouse) with just a single rabbitmq server installed directly on the cloud controller (meaning: no nova-compute). I never have any rabbit issues in that cloud. </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
<br class="">
_______________________________________________<br class="">
OpenStack-operators mailing list<br class="">
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank" class="">OpenStack-operators@lists.openstack.org</a><br class="">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</span></div>
</div>
</span>
</div>
<br class="">_______________________________________________<br class="">
OpenStack-operators mailing list<br class="">
<a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br class="">
<br class=""></blockquote></div>
_______________________________________________<br class="">OpenStack-operators mailing list<br class=""><a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<br class=""></div></blockquote></div><br class=""></div></body></html>