Le mar. 6 août 2019 à 17:14, Ben Nemec <openstack@nemebean.com> a écrit :

Another thing to check if you're having seemingly inexplicable messaging
issues is that there isn't a notification queue filling up somewhere. If
notifications are enabled somewhere but nothing is consuming them the
size of the queue will eventually grind rabbit to a halt.

I used to check queue sizes through the rabbit web ui, so I have to
admit I'm not sure how to do it through the cli.

You can use the following command to monitor your queues and observe size and growing:

```

watch -c "rabbitmqctl list_queues name messages_unacknowledged"

```

Or also something like that:

```

rabbitmqctl list_queues messages consumers name message_bytes messages_unacknowledged > messages_ready head_message_timestamp consumer_utilisation memory state | grep reply

```

 

On 7/31/19 10:48 AM, Gabriele Santomaggio wrote:
> Hi,
> Are you using ssl connections ?
>
> Can be this issue ?
> https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1800957
>
>
> ------------------------------------------------------------------------
> *From:* Laurent Dumont <laurentfdumont@gmail.com>
> *Sent:* Wednesday, July 31, 2019 4:20 PM
> *To:* Grant Morley
> *Cc:* openstack-operators@lists.openstack.org
> *Subject:* Re: Slow instance launch times due to RabbitMQ
> That is a bit strange, list_queues should return stuff. Couple of ideas :
>
>   * Are the Rabbit connection failure logs on the compute pointing to a
>     specific controller?
>   * Are there any logs within Rabbit on the controller that would point
>     to a transient issue?
>   * cluster_status is a snapshot of the cluster at the time you ran the
>     command. If the alarms have cleared, you won't see anything.
>   * If you have the RabbitMQ management plugin activated, I would
>     recommend a quick look to see the historical metrics and overall status.
>
>
> On Wed, Jul 31, 2019 at 9:35 AM Grant Morley <grant@civo.com
> <mailto:grant@civo.com>> wrote:
>
>     Hi guys,
>
>     We are using Ubuntu 16 and OpenStack ansible to do our setup.
>
>     rabbitmqctl list_queues
>     Listing queues
>
>     (Doesn't appear to be any queues )
>
>     rabbitmqctl cluster_status
>
>     Cluster status of node
>     'rabbit@management-1-rabbit-mq-container-b4d7791f'
>     [{nodes,[{disc,['rabbit@management-1-rabbit-mq-container-b4d7791f',
>                      'rabbit@management-2-rabbit-mq-container-b455e77d',
>                      'rabbit@management-3-rabbit-mq-container-1d6ae377']}]},
>       {running_nodes,['rabbit@management-3-rabbit-mq-container-1d6ae377',
>                       'rabbit@management-2-rabbit-mq-container-b455e77d',
>                       'rabbit@management-1-rabbit-mq-container-b4d7791f']},
>       {cluster_name,<<"openstack">>},
>       {partitions,[]},
>       {alarms,[{'rabbit@management-3-rabbit-mq-container-1d6ae377',[]},
>                {'rabbit@management-2-rabbit-mq-container-b455e77d',[]},
>                {'rabbit@management-1-rabbit-mq-container-b4d7791f',[]}]}]
>
>     Regards,
>
>     On 31/07/2019 11:49, Laurent Dumont wrote:
>>     Could you forward the output of the following commands on a
>>     controller node? :
>>
>>     rabbitmqctl cluster_status
>>     rabbitmqctl list_queues
>>
>>     You won't necessarily see a high load on a Rabbit cluster that is
>>     in a bad state.
>>
>>     On Wed, Jul 31, 2019 at 5:19 AM Grant Morley <grant@civo.com
>>     <mailto:grant@civo.com>> wrote:
>>
>>         Hi all,
>>
>>         We are randomly seeing slow instance launch / deletion times
>>         and it appears to be because of RabbitMQ. We are seeing a lot
>>         of these messages in the logs for Nova and Neutron:
>>
>>         ERROR oslo.messaging._drivers.impl_rabbit [-]
>>         [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on
>>         10.6.2.212:5671 <http://10.6.2.212:5671> is unreachable: Too
>>         many heartbeats missed. Trying again in 1 seconds. Client
>>         port: 37098: ConnectionForced: Too many heartbeats missed
>>
>>         The RabbitMQ cluster isn't under high load and I am not seeing
>>         any packets drop over the network when I do some tracing.
>>
>>         We are only running 15 compute nodes currently and have >1000
>>         instances so it isn't a large deployment.
>>
>>         Are there any good configuration tweaks for RabbitMQ running
>>         on OpenStack Queens?
>>
>>         Many Thanks,
>>
>>         --
>>
>>         Grant Morley
>>         Cloud Lead, Civo Ltd
>>         www.civo.com <https://www.civo.com/>| Signup for an account!
>>         <https://www.civo.com/signup>
>>
>     --
>
>     Grant Morley
>     Cloud Lead, Civo Ltd
>     www.civo.com <https://www.civo.com/>| Signup for an account!
>     <https://www.civo.com/signup>
>

--

Hervé Beraud

Senior Software Engineer

Red Hat - Openstack Oslo

irc: hberaud

-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----