<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>If you tune rabbit then for: <br>
</p>
<p><span style="color:rgb(80,0,80)">heartbeat_timeout_threshold = 0</span></p>
<p>That should help with the error message you are getting. <br>
</p>
<p>That is a lot of messages queued. We had the same because we were
not using ceilometer but had the "notifications" still turned on
for it for services.</p>
<p>Are all of the ready messages for "notifications.info" for the
various services ( Nova, Neutron, Keystone etc )</p>
<p>If that is the case you can disable those messages in your config
files for those services. Look for:</p>
<p><span style="color:rgb(80,0,80)"># Notifications<br>
[oslo_messaging_notifications]<br>
notification_topics = notifications<br>
driver = noop</span></p>
<p><span style="color:rgb(80,0,80)"><font color="#000000">Make sure
the driver option is set to "noop" by default it will be set
too "messagingv2" and then restart the service and that should
stop sending messages to the queue. You can then purge the
"notifications.info" queue for Nova or Neutron etc..<br>
</font></span></p>
<p>We only had the "message ready" when we had the setting for
ceilometer set but was not using it. Also only purge a queue if it
is for that reason. Do not purge the queue if it is for any other
reason than that as it can cause issues.</p>
<p>Hope that helps.</p>
<p>Grant<br>
<span style="color:rgb(80,0,80)"><font color="#000000"></font></span></p>
<div class="moz-cite-prefix">On 20/03/2020 14:38, Satish Patel
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAPgF-foBsBuhiVtHW_ekAYEwGx_PquwmDowKNbNZVEf3y4i1yA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Grant, </div>
<div><br>
</div>
<div>But i am seeing lots of following logs on my compute nodes
running stein release. </div>
<div><br>
</div>
<div>2020-03-20 10:34:46.132 53425 WARNING
oslo.messaging._drivers.impl_rabbit [-] Unexpected error
during heartbeart thread processing, retrying...:
ConnectionForced: Too many heartbeats missed.<br>
</div>
<div><br>
</div>
<div>Find attached screenshot of one of rabbitMQ node, i have
lots of messages in "Ready: 15339" is this looks normal to
you?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Mar 20, 2020 at 4:55
AM Grant Morley <<a href="mailto:grant@civo.com"
moz-do-not-send="true">grant@civo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi,</p>
<p>There was a bug in Queens that meant there was an issue
with the heartbeat timeouts. Setting it to 0 gets around
that bug. I believe that was fixed in Rocky and above, so
your Stein installation should be fine. <br>
</p>
<p>Setting the value to 0 For us meant we stopped getting
errors in the logs for:</p>
<p>"Too many heartbeats missed, trying to force connect to
RabbitMQ"</p>
<p>Regards,<br>
</p>
<div>On 19/03/2020 18:53, Satish Patel wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I have question related following setting,
why are you disabling heartbeat timeout?
<div><br>
</div>
<div><span style="color:rgb(80,0,80)">heartbeat_timeout_threshold
= 0 </span><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Mar 19, 2020
at 1:32 PM Satish Patel <<a
href="mailto:satish.txt@gmail.com" target="_blank"
moz-do-not-send="true">satish.txt@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Great, thanks! Did you guys tune your
nova component for rabbitMQ? <br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Mar 19,
2020 at 1:26 PM Grant Morley <<a
href="mailto:grant@civo.com" target="_blank"
moz-do-not-send="true">grant@civo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>We left ours on the default value of 1 and
that still seems to be fine.</p>
<p>Grant<br>
</p>
<div>On 19/03/2020 17:13, Satish Patel wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">how about rpc_worker ?
<div> </div>
<div>currently i have rpc_worker=1 </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu,
Mar 19, 2020 at 1:02 PM Grant Morley <<a
href="mailto:grant@civo.com"
target="_blank" moz-do-not-send="true">grant@civo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Correct, you need to add:</p>
<p><span style="color:rgb(80,0,80)">>
> heartbeat_timeout_threshold = 0</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_conn_pool_size = 300</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_thread_pool_size = 2048</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_response_timeout = 3600</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_poll_timeout = 60</span></p>
<p>To your Neutron nodes</p>
<p>And you can add:</p>
<p><br>
<span style="color:rgb(80,0,80)"><span
style="color:rgb(80,0,80)">>>
executor_thread_pool_size = 64</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>>
rpc_response_timeout = 3600</span></span></p>
<p><span style="color:rgb(80,0,80)"><span
style="color:rgb(80,0,80)"><font
color="#000000">To your compute
nodes (neutron.conf) However I
found just adding the changes to
the neturon servers really
helped.</font></span></span></p>
<p><span style="color:rgb(80,0,80)"><span
style="color:rgb(80,0,80)"><font
color="#000000">I would
recommend just starting with
your neutron nodes first to see
if that helps. If you find your
compute nodes are still having
issues then change the settings
on those after.</font></span></span></p>
<p><span style="color:rgb(80,0,80)"><span
style="color:rgb(80,0,80)"><font
color="#000000">Regards,</font><br>
</span></span></p>
<span style="color:rgb(80,0,80)"></span>
<div>On 19/03/2020 16:53, Satish Patel
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I am running
openstack-ansible (Queens / Stein
both) so this is what i am going
to do, am i doing correctly?
<div><br>
</div>
<div>neutron-server (container) I
have 3 neutron node. </div>
<div><span
style="color:rgb(80,0,80)">>
> heartbeat_timeout_threshold
= 0</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_conn_pool_size = 300</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_thread_pool_size = 2048</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_response_timeout = 3600</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>
> rpc_poll_timeout = 60</span><br>
</div>
<div><br>
</div>
<div>330 compute nodes (agent
neutron.conf) going to add
following:</div>
<div><span
style="color:rgb(80,0,80)">>>
executor_thread_pool_size = 64</span><br
style="color:rgb(80,0,80)">
<span style="color:rgb(80,0,80)">>>
rpc_response_timeout = 3600</span><br>
</div>
<div><span
style="color:rgb(80,0,80)"><br>
</span></div>
<div><br>
</div>
<div><br>
</div>
<div>How about nova? should i be
doing that on nova as well to
reduce load on rabbitMQ?</div>
<div><span
style="color:rgb(80,0,80)"><br>
</span></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Thu, Mar 19, 2020 at 12:35 PM
Grant Morley <<a
href="mailto:grant@civo.com"
target="_blank"
moz-do-not-send="true">grant@civo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Satish,</p>
<p>You will need to add those to
the "neutron.conf" file on
your network nodes. If you are
running OS-A I would do it on
your "neutron-server" nodes
and add the following to your
agents containers:</p>
<p>executor_thread_pool_size =
64<br>
rpc_response_timeout = 3600<br>
</p>
<p>Regards,<br>
</p>
<div>On 19/03/2020 16:27, Satish
Patel wrote:<br>
</div>
<blockquote type="cite">
<pre>Erik,
If i want to adopt following setting then where i should add them in
Queens openstack, neutron-server or all my compute nodes? which
setting will go where?
heartbeat_timeout_threshold = 0
rpc_conn_pool_size = 300
rpc_thread_pool_size = 2048
rpc_response_timeout = 3600
rpc_poll_timeout = 60
## Rpc all
executor_thread_pool_size = 64
rpc_response_timeout = 3600
On Wed, Mar 11, 2020 at 9:05 PM Erik Olof Gunnar Andersson
<a href="mailto:eandersson@blizzard.com" target="_blank" moz-do-not-send="true"><eandersson@blizzard.com></a> wrote:
</pre>
<blockquote type="cite">
<pre>We are hitting something awfully similar.
We have basically been hitting a few pretty serious bugs with RabbitMQ.
The main one is when a RabbitMQ server crashes, or gets split brain it does not always recover, or even when just one node is restarted. We sometimes end up with orphaned consumers that keep consuming messages, but goes to /dev/null pretty much. Another issue is that sometimes bindings stop working. They are visually there, but simply does not route traffic to the intended queues.
e.g. <a href="https://github.com/rabbitmq/rabbitmq-server/issues/641" target="_blank" moz-do-not-send="true">https://github.com/rabbitmq/rabbitmq-server/issues/641</a>
I wrote two quick scripts to audit these issues.
<a href="http://paste.openstack.org/show/790569/" target="_blank" moz-do-not-send="true">http://paste.openstack.org/show/790569/</a> - Check if you have orphaned consumers (may need pagination if you have a large deployment).
<a href="http://paste.openstack.org/show/790570/" target="_blank" moz-do-not-send="true">http://paste.openstack.org/show/790570/</a> - Check if the bindings are bad for a specific queue.
The main issue seems to be the number of queues + connections causing the recovery after restarting a node to cause bindings and/or queues to get into an "orphaned" state.
Best Regards, Erik Olof Gunnar Andersson
-----Original Message-----
From: Satish Patel <a href="mailto:satish.txt@gmail.com" target="_blank" moz-do-not-send="true"><satish.txt@gmail.com></a>
Sent: Wednesday, March 11, 2020 5:14 PM
To: Grant Morley <a href="mailto:grant@civo.com" target="_blank" moz-do-not-send="true"><grant@civo.com></a>
Cc: <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank" moz-do-not-send="true">openstack-discuss@lists.openstack.org</a>
Subject: Re: Neutron RabbitMQ issues
I am also dealing with some short of rabbitmq performance issue but its not as worst you your issue.
This is my favorite video, not sure you have seen this before or not but anyway posting here - <a href="https://urldefense.com/v3/__https://www.youtube.com/watch?v=bpmgxrPOrZw__;!!Ci6f514n9QsL8ck!1rOR_L7ya6zmMgZ0owpfO7NvhsPOzbgyUplonob2awcg8hd80yCAT_ynvarUEZv4Mw$" target="_blank" moz-do-not-send="true">https://urldefense.com/v3/__https://www.youtube.com/watch?v=bpmgxrPOrZw__;!!Ci6f514n9QsL8ck!1rOR_L7ya6zmMgZ0owpfO7NvhsPOzbgyUplonob2awcg8hd80yCAT_ynvarUEZv4Mw$</a>
On Wed, Mar 11, 2020 at 10:24 AM Grant Morley <a href="mailto:grant@civo.com" target="_blank" moz-do-not-send="true"><grant@civo.com></a> wrote:
</pre>
<blockquote type="cite">
<pre>Hi all,
We are currently experiencing some fairly major issues with our
OpenStack cluster. It all appears to be with Neutron and RabbitMQ. We
are seeing a lot of time out messages in responses to replies and
because of this instance creation or anything to do with instances and
networking is broken.
We are running OpenStack Queens.
We have already tuned Rabbit for Neutron by doing the following on neutron:
heartbeat_timeout_threshold = 0
rpc_conn_pool_size = 300
rpc_thread_pool_size = 2048
rpc_response_timeout = 3600
rpc_poll_timeout = 60
## Rpc all
executor_thread_pool_size = 64
rpc_response_timeout = 3600
What we are seeing in the error logs for neutron for all services
(l3-agent, dhcp, linux-bridge etc ) are these timeouts:
<a href="https://urldefense.com/v3/__https://pastebin.com/Fjh23A5a__;!!Ci6f514n" target="_blank" moz-do-not-send="true">https://urldefense.com/v3/__https://pastebin.com/Fjh23A5a__;!!Ci6f514n</a>
9QsL8ck!1rOR_L7ya6zmMgZ0owpfO7NvhsPOzbgyUplonob2awcg8hd80yCAT_ynvapLQK
9aOA$
We have manually tried to get everything in sync by forcing fail-over
of the networking which seems to get routers in sync.
We are also seeing that there are a lot of "unacknowledged" messages
in RabbitMQ for 'q-plugin' in the neutron queues.
Some times restarting of the services on neutron gets these back
acknowledged again, however the timeouts come back.
The RabbitMQ servers themselves are not loaded at all. All memory,
file descriptors and errlang processes have plenty of resources available.
We are also seeing a lot of rpc issues:
Timeout in RPC method release_dhcp_port. Waiting for 1523 seconds
before next attempt. If the server is not down, consider increasing
the rpc_response_timeout option as Neutron server(s) may be overloaded
and unable to respond quickly enough.: MessagingTimeout: Timed out
waiting for a reply to message ID 965fa44ab4f6462fa378a1cf7259aad4
2020-03-10 19:02:33.548 16242 ERROR neutron.common.rpc
[req-a858afbb-5083-4e21-a309-6ee53582c4d9 - - - - -] Timeout in RPC
method release_dhcp_port. Waiting for 3347 seconds before next attempt.
If the server is not down, consider increasing the
rpc_response_timeout option as Neutron server(s) may be overloaded and
unable to respond quickly enough.: MessagingTimeout: Timed out waiting
for a reply to message ID 7937465f15634fbfa443fe1758a12a9c
Does anyone know if there is anymore tuning to be done at all?
Upgrading for us at the moment to a newer version isn't really an
option unfortunately.
Because of our setup, we also have roughly 800 routers enabled and I
know that will be putting a load on the system. However these problems
have only started to happen roughly 1 week ago and have steadily got worse.
If anyone has any use cases for this or any more recommendations that
would be great.
Many thanks,
</pre>
</blockquote>
</blockquote>
</blockquote>
<div>-- <br>
<div><br>
</div>
<div>
<div
style="font-family:helvetica,Arial;font-size:13px"><font
face="Lucida Grande,
Verdana, Arial,
sans-serif"
color="#A8D8FF"><span
style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img
src="https://www.civo.com/images/email-logo.jpg" moz-do-not-send="true"></span></font>
<div
style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span>Grant
Morley</span> <font
face="Hind, Lucida
Grande, Verdana,
Arial, sans-serif"><span
style="font-size:15px"></span></font></div>
<div
style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font
face="Hind, Lucida
Grande, Verdana,
Arial, sans-serif"><span
style="font-size:15px">Cloud Lead, Civo Ltd</span></font></div>
<div
style="font-family:Hind,helvetica,arial;margin-left:90px"><a
href="https://www.civo.com/" target="_blank" moz-do-not-send="true">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup" target="_blank"
moz-do-not-send="true">Signup
for an account!</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div>-- <br>
<div><br>
</div>
<div>
<div
style="font-family:helvetica,Arial;font-size:13px"><font
face="Lucida Grande, Verdana,
Arial, sans-serif"
color="#A8D8FF"><span
style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img
src="https://www.civo.com/images/email-logo.jpg" moz-do-not-send="true"></span></font>
<div
style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span>Grant
Morley</span> <font
face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"><span
style="font-size:15px"></span></font></div>
<div
style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font
face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"><span
style="font-size:15px">Cloud
Lead, Civo Ltd</span></font></div>
<div
style="font-family:Hind,helvetica,arial;margin-left:90px"><a
href="https://www.civo.com/"
target="_blank"
moz-do-not-send="true">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup" target="_blank"
moz-do-not-send="true">Signup
for an account!</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div>-- <br>
<div><br>
</div>
<div>
<div
style="font-family:helvetica,Arial;font-size:13px"><font
face="Lucida Grande, Verdana, Arial,
sans-serif" color="#A8D8FF"><span
style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img
src="https://www.civo.com/images/email-logo.jpg" moz-do-not-send="true"></span></font>
<div
style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span>Grant
Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial,
sans-serif"><span
style="font-size:15px"></span></font></div>
<div
style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font
face="Hind, Lucida Grande, Verdana,
Arial, sans-serif"><span
style="font-size:15px">Cloud Lead,
Civo Ltd</span></font></div>
<div
style="font-family:Hind,helvetica,arial;margin-left:90px"><a
href="https://www.civo.com/"
target="_blank" moz-do-not-send="true">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup"
target="_blank" moz-do-not-send="true">Signup
for an account!</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<div>-- <br>
<div><br>
</div>
<div>
<div style="font-family:helvetica,Arial;font-size:13px"><font
face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"><span
style="font-size:11px;line-height:15px;float:left;margin-right:10px"><img
src="https://www.civo.com/images/email-logo.jpg"
moz-do-not-send="true"></span></font>
<div
style="margin-left:90px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"><span
style="font-family:Hind,"Lucida
Grande",Verdana,Arial,sans-serif;font-size:18px;color:rgb(35,157,255)">Grant
Morley</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"><span
style="font-size:15px"></span></font></div>
<div
style="margin-left:90px;line-height:15px;color:rgb(136,136,136);margin-bottom:10px"><font
face="Hind, Lucida Grande, Verdana, Arial,
sans-serif"><span style="font-size:15px">Cloud
Lead, Civo Ltd</span></font></div>
<div
style="font-family:Hind,helvetica,arial;margin-left:90px"><a
href="https://www.civo.com/"
style="color:rgb(35,157,255);font-family:Hind,"Lucida
Grande",Verdana,Arial,sans-serif;line-height:15px;text-decoration:none"
target="_blank" moz-do-not-send="true">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup"
style="color:rgb(35,157,255);font-family:Hind,"Lucida
Grande",Verdana,Arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none"
target="_blank" moz-do-not-send="true">Signup for
an account!</a></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<div><br>
</div>
<div>
<div style="font-family: helvetica, Arial; font-size: 13px;"><font
face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"><span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"><img
src="https://www.civo.com/images/email-logo.jpg"
style="width: 70px; color: #a8d8ff; font-family: 'Lucida
Grande', Verdana, Arial, sans-serif; font-size: 11px;
line-height: 15px; float: left; margin-top: 0px;
margin-right: 10px"></span></font>
<div style="margin-left: 90px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"><span
style="font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; font-size: 18px; color:
rgb(35, 157, 255);">Grant Morley</span> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;"></span></font></div>
<div style="margin-left: 90px; line-height: 15px; color:
#888888; margin-bottom: 10px;"><font face="Hind, Lucida
Grande, Verdana, Arial, sans-serif"><span
style="font-size: 15px;">Cloud Lead, Civo Ltd</span></font></div>
<div style="font-family: Hind, helvetica, arial; margin-left:
90px;"><a href="https://www.civo.com/" style="color: rgb(35,
157, 255); font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; line-height: 15px;
text-decoration: none;">www.civo.com</a><font
color="#888888"> | </font><a
href="https://www.civo.com/signup" style="color: rgb(35,
157, 255); font-family: Hind, "Lucida Grande",
Verdana, Arial, sans-serif; line-height: 15px;
margin-right: 6px; text-decoration: none;">Signup for an
account!</a></div>
</div>
</div>
</div>
</body>
</html>