<div dir="ltr">That is a bit strange, list_queues should return stuff. Couple of ideas :<div><ul><li>Are the Rabbit connection failure logs on the compute pointing to a specific controller?</li><li>Are there any logs within Rabbit on the controller that would point to a transient issue?</li><li>cluster_status is a snapshot of the cluster at the time you ran the command. If the alarms have cleared, you won't see anything.</li><li>If you have the RabbitMQ management plugin activated, I would recommend a quick look to see the historical metrics and overall status.</li></ul></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 9:35 AM Grant Morley <<a href="mailto:grant@civo.com">grant@civo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi guys,</p>
<p>We are using Ubuntu 16 and OpenStack ansible to do our setup.</p>
<p>rabbitmqctl list_queues<br>
Listing queues<br>
</p>
<p>(Doesn't appear to be any queues )</p>
<p>rabbitmqctl cluster_status</p>
<p>Cluster status of node
'rabbit@management-1-rabbit-mq-container-b4d7791f'<br>
[{nodes,[{disc,['rabbit@management-1-rabbit-mq-container-b4d7791f',<br>
'rabbit@management-2-rabbit-mq-container-b455e77d',<br>
'rabbit@management-3-rabbit-mq-container-1d6ae377']}]},<br>
{running_nodes,['rabbit@management-3-rabbit-mq-container-1d6ae377',<br>
'rabbit@management-2-rabbit-mq-container-b455e77d',<br>
'rabbit@management-1-rabbit-mq-container-b4d7791f']},<br>
{cluster_name,<<"openstack">>},<br>
{partitions,[]},<br>
{alarms,[{'rabbit@management-3-rabbit-mq-container-1d6ae377',[]},<br>
{'rabbit@management-2-rabbit-mq-container-b455e77d',[]},<br>
{'rabbit@management-1-rabbit-mq-container-b4d7791f',[]}]}]</p>
<p>Regards,<br>
</p>
<div class="gmail-m_-173761228863317427moz-cite-prefix">On 31/07/2019 11:49, Laurent Dumont
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could you forward the output of the following
commands on a controller node? :
<div><br>
</div>
<div>rabbitmqctl cluster_status</div>
<div>rabbitmqctl list_queues</div>
<div><br>
</div>
<div>You won't necessarily see a high load on a Rabbit cluster
that is in a bad state.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jul 31, 2019 at 5:19
AM Grant Morley <<a href="mailto:grant@civo.com" target="_blank">grant@civo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi all,</p>
<p>We are randomly seeing slow instance launch / deletion
times and it appears to be because of RabbitMQ. We are
seeing a lot of these messages in the logs for Nova and
Neutron:</p>
<p>ERROR oslo.messaging._drivers.impl_rabbit [-]
[f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on <a href="http://10.6.2.212:5671" target="_blank">10.6.2.212:5671</a> is
unreachable: Too many heartbeats missed. Trying again in 1
seconds. Client port: 37098: ConnectionForced: Too many
heartbeats missed</p>
<p>The RabbitMQ cluster isn't under high load and I am not
seeing any packets drop over the network when I do some
tracing.</p>
<p>We are only running 15 compute nodes currently and have
>1000 instances so it isn't a large deployment.<br>
</p>
<p>Are there any good configuration tweaks for RabbitMQ
running on OpenStack Queens? </p>
<p>Many Thanks,<br>
</p>
<div class="gmail-m_-173761228863317427gmail-m_4478358746559024905moz-signature">-- <br>
<div><br>
</div>
<div>
<div style="font-family:helvetica,Arial;font-size:13px"><font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"><span style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img src="https://www.civo.com/images/email-logo.jpg"></span></font>
<div style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span>Grant
Morley</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"><span style="font-size:15px"></span></font></div>
<div style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font face="Hind, Lucida Grande, Verdana, Arial,
sans-serif"><span style="font-size:15px">Cloud
Lead, Civo Ltd</span></font></div>
<div style="font-family:Hind,helvetica,arial;margin-left:90px"><a href="https://www.civo.com/" target="_blank">www.civo.com</a><font color="#888888"> | </font><a href="https://www.civo.com/signup" target="_blank">Signup for
an account!</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div class="gmail-m_-173761228863317427moz-signature">-- <br>
<div><br>
</div>
<div>
<div style="font-family:helvetica,Arial;font-size:13px"><font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"><span style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img src="https://www.civo.com/images/email-logo.jpg"></span></font>
<div style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span style="font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;font-size:18px;color:rgb(35,157,255)">Grant Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"><span style="font-size:15px"></span></font></div>
<div style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span style="font-size:15px">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family:Hind,helvetica,arial;margin-left:90px"><a href="https://www.civo.com/" style="color:rgb(35,157,255);font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;line-height:15px;text-decoration:none" target="_blank">www.civo.com</a><font color="#888888"> | </font><a href="https://www.civo.com/signup" style="color:rgb(35,157,255);font-family:Hind,"Lucida Grande",Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">Signup for an
account!</a></div>
</div>
</div>
</div>
</div>
</blockquote></div>