Re: Slow instance launch times due to RabbitMQ

7 Aug 2019

      Le mar. 6 août 2019 à 17:14, Ben Nemec <openstack@nemebean.com> a écrit :
...
Another thing to check if you're having seemingly inexplicable messaging
issues is that there isn't a notification queue filling up somewhere. If
notifications are enabled somewhere but nothing is consuming them the
size of the queue will eventually grind rabbit to a halt.
I used to check queue sizes through the rabbit web ui, so I have to
admit I'm not sure how to do it through the cli.
You can use the following command to monitor your queues and observe size
and growing:

```
watch -c "rabbitmqctl list_queues name messages_unacknowledged"
```

Or also something like that:

```
rabbitmqctl list_queues messages consumers name message_bytes
messages_unacknowledged > messages_ready head_message_timestamp
consumer_utilisation memory state | grep reply
```
...
On 7/31/19 10:48 AM, Gabriele Santomaggio wrote:
...
Hi,
Are you using ssl connections ?
Can be this issue ?
https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1800957
------------------------------------------------------------------------
*From:* Laurent Dumont <laurentfdumont@gmail.com>
*Sent:* Wednesday, July 31, 2019 4:20 PM
*To:* Grant Morley
*Cc:* openstack-operators@lists.openstack.org
*Subject:* Re: Slow instance launch times due to RabbitMQ
That is a bit strange, list_queues should return stuff. Couple of ideas :
* Are the Rabbit connection failure logs on the compute pointing to a
    specific controller?
  * Are there any logs within Rabbit on the controller that would point
    to a transient issue?
  * cluster_status is a snapshot of the cluster at the time you ran the
    command. If the alarms have cleared, you won't see anything.
  * If you have the RabbitMQ management plugin activated, I would
    recommend a quick look to see the historical metrics and overall
status.
On Wed, Jul 31, 2019 at 9:35 AM Grant Morley <grant@civo.com
<mailto:grant@civo.com>> wrote:
Hi guys,
We are using Ubuntu 16 and OpenStack ansible to do our setup.
rabbitmqctl list_queues
    Listing queues
(Doesn't appear to be any queues )
rabbitmqctl cluster_status
Cluster status of node
    'rabbit@management-1-rabbit-mq-container-b4d7791f'
    [{nodes,[{disc,['rabbit@management-1-rabbit-mq-container-b4d7791f',
                     'rabbit@management-2-rabbit-mq-container-b455e77d',
                     'rabbit@management-3-rabbit-mq-container-1d6ae377
']}]},
      {running_nodes,['rabbit@management-3-rabbit-mq-container-1d6ae377
',
                      'rabbit@management-2-rabbit-mq-container-b455e77d
',
                      'rabbit@management-1-rabbit-mq-container-b4d7791f
']},
      {cluster_name,<<"openstack">>},
      {partitions,[]},
      {alarms,[{'rabbit@management-3-rabbit-mq-container-1d6ae377',[]},
               {'rabbit@management-2-rabbit-mq-container-b455e77d',[]},
               {'rabbit@management-1-rabbit-mq-container-b4d7791f
',[]}]}]
Regards,
On 31/07/2019 11:49, Laurent Dumont wrote:
...
Could you forward the output of the following commands on a
    controller node? :
rabbitmqctl cluster_status
    rabbitmqctl list_queues
You won't necessarily see a high load on a Rabbit cluster that is
    in a bad state.
On Wed, Jul 31, 2019 at 5:19 AM Grant Morley <grant@civo.com
    <mailto:grant@civo.com>> wrote:
Hi all,
We are randomly seeing slow instance launch / deletion times
        and it appears to be because of RabbitMQ. We are seeing a lot
        of these messages in the logs for Nova and Neutron:
ERROR oslo.messaging._drivers.impl_rabbit [-]
        [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on
        10.6.2.212:5671 <http://10.6.2.212:5671> is unreachable: Too
        many heartbeats missed. Trying again in 1 seconds. Client
        port: 37098: ConnectionForced: Too many heartbeats missed
The RabbitMQ cluster isn't under high load and I am not seeing
        any packets drop over the network when I do some tracing.
We are only running 15 compute nodes currently and have >1000
        instances so it isn't a large deployment.
Are there any good configuration tweaks for RabbitMQ running
        on OpenStack Queens?
Many Thanks,
--
Grant Morley
        Cloud Lead, Civo Ltd
        www.civo.com <https://www.civo.com/>| Signup for an account!
        <https://www.civo.com/signup>
--
Grant Morley
    Cloud Lead, Civo Ltd
    www.civo.com <https://www.civo.com/>| Signup for an account!
    <https://www.civo.com/signup>
-- 
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----

Re: Slow instance launch times due to RabbitMQ

Herve Beraud