<div dir="ltr">Could you forward the output of the following commands on a controller node? :<div><br></div><div>rabbitmqctl cluster_status</div><div>rabbitmqctl list_queues</div><div><br></div><div>You won't necessarily see a high load on a Rabbit cluster that is in a bad state.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 5:19 AM Grant Morley <<a href="mailto:grant@civo.com">grant@civo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi all,</p>
<p>We are randomly seeing slow instance launch / deletion times and
it appears to be because of RabbitMQ. We are seeing a lot of these
messages in the logs for Nova and Neutron:</p>
<p>ERROR oslo.messaging._drivers.impl_rabbit [-]
[f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on
<a href="http://10.6.2.212:5671" target="_blank">10.6.2.212:5671</a> is unreachable: Too many heartbeats missed. Trying
again in 1 seconds. Client port: 37098: ConnectionForced: Too many
heartbeats missed</p>
<p>The RabbitMQ cluster isn't under high load and I am not seeing
any packets drop over the network when I do some tracing.</p>
<p>We are only running 15 compute nodes currently and have >1000
instances so it isn't a large deployment.<br>
</p>
<p>Are there any good configuration tweaks for RabbitMQ running on
OpenStack Queens? </p>
<p>Many Thanks,<br>
</p>
<div class="gmail-m_4478358746559024905moz-signature">-- <br>
<div><br>
</div>
<div>
<div style="font-family:helvetica,Arial;font-size:13px"><font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"><span style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img src="https://www.civo.com/images/email-logo.jpg"></span></font>
<div style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span style="font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;font-size:18px;color:rgb(35,157,255)">Grant Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"><span style="font-size:15px"></span></font></div>
<div style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span style="font-size:15px">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family:Hind,helvetica,arial;margin-left:90px"><a href="https://www.civo.com/" style="color:rgb(35,157,255);font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;line-height:15px;text-decoration:none" target="_blank">www.civo.com</a><font color="#888888"> | </font><a href="https://www.civo.com/signup" style="color:rgb(35,157,255);font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">Signup for an
account!</a></div>
</div>
</div>
</div>
</div>
</blockquote></div>