Open Stack

Le mar. 6 août 2019 à 17:14, Ben Nemec <openstack at nemebean.com> a écrit :

> Another thing to check if you're having seemingly inexplicable messaging
> issues is that there isn't a notification queue filling up somewhere. If
> notifications are enabled somewhere but nothing is consuming them the
> size of the queue will eventually grind rabbit to a halt.
>
> I used to check queue sizes through the rabbit web ui, so I have to
> admit I'm not sure how to do it through the cli.
>

You can use the following command to monitor your queues and observe size
and growing:

```
watch -c "rabbitmqctl list_queues name messages_unacknowledged"
```

Or also something like that:

```
rabbitmqctl list_queues messages consumers name message_bytes
messages_unacknowledged > messages_ready head_message_timestamp
consumer_utilisation memory state | grep reply
```


>
> On 7/31/19 10:48 AM, Gabriele Santomaggio wrote:
> > Hi,
> > Are you using ssl connections ?
> >
> > Can be this issue ?
> > https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1800957
> >
> >
> > ------------------------------------------------------------------------
> > *From:* Laurent Dumont <laurentfdumont at gmail.com>
> > *Sent:* Wednesday, July 31, 2019 4:20 PM
> > *To:* Grant Morley
> > *Cc:* openstack-operators at lists.openstack.org
> > *Subject:* Re: Slow instance launch times due to RabbitMQ
> > That is a bit strange, list_queues should return stuff. Couple of ideas :
> >
> >   * Are the Rabbit connection failure logs on the compute pointing to a
> >     specific controller?
> >   * Are there any logs within Rabbit on the controller that would point
> >     to a transient issue?
> >   * cluster_status is a snapshot of the cluster at the time you ran the
> >     command. If the alarms have cleared, you won't see anything.
> >   * If you have the RabbitMQ management plugin activated, I would
> >     recommend a quick look to see the historical metrics and overall
> status.
> >
> >
> > On Wed, Jul 31, 2019 at 9:35 AM Grant Morley <grant at civo.com
> > <mailto:grant at civo.com>> wrote:
> >
> >     Hi guys,
> >
> >     We are using Ubuntu 16 and OpenStack ansible to do our setup.
> >
> >     rabbitmqctl list_queues
> >     Listing queues
> >
> >     (Doesn't appear to be any queues )
> >
> >     rabbitmqctl cluster_status
> >
> >     Cluster status of node
> >     'rabbit at management-1-rabbit-mq-container-b4d7791f'
> >     [{nodes,[{disc,['rabbit at management-1-rabbit-mq-container-b4d7791f',
> >                      'rabbit at management-2-rabbit-mq-container-b455e77d',
> >                      'rabbit at management-3-rabbit-mq-container-1d6ae377
> ']}]},
> >       {running_nodes,['rabbit at management-3-rabbit-mq-container-1d6ae377
> ',
> >                       'rabbit at management-2-rabbit-mq-container-b455e77d
> ',
> >                       'rabbit at management-1-rabbit-mq-container-b4d7791f
> ']},
> >       {cluster_name,<<"openstack">>},
> >       {partitions,[]},
> >       {alarms,[{'rabbit at management-3-rabbit-mq-container-1d6ae377',[]},
> >                {'rabbit at management-2-rabbit-mq-container-b455e77d',[]},
> >                {'rabbit at management-1-rabbit-mq-container-b4d7791f
> ',[]}]}]
> >
> >     Regards,
> >
> >     On 31/07/2019 11:49, Laurent Dumont wrote:
> >>     Could you forward the output of the following commands on a
> >>     controller node? :
> >>
> >>     rabbitmqctl cluster_status
> >>     rabbitmqctl list_queues
> >>
> >>     You won't necessarily see a high load on a Rabbit cluster that is
> >>     in a bad state.
> >>
> >>     On Wed, Jul 31, 2019 at 5:19 AM Grant Morley <grant at civo.com
> >>     <mailto:grant at civo.com>> wrote:
> >>
> >>         Hi all,
> >>
> >>         We are randomly seeing slow instance launch / deletion times
> >>         and it appears to be because of RabbitMQ. We are seeing a lot
> >>         of these messages in the logs for Nova and Neutron:
> >>
> >>         ERROR oslo.messaging._drivers.impl_rabbit [-]
> >>         [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on
> >>         10.6.2.212:5671 <http://10.6.2.212:5671> is unreachable: Too
> >>         many heartbeats missed. Trying again in 1 seconds. Client
> >>         port: 37098: ConnectionForced: Too many heartbeats missed
> >>
> >>         The RabbitMQ cluster isn't under high load and I am not seeing
> >>         any packets drop over the network when I do some tracing.
> >>
> >>         We are only running 15 compute nodes currently and have >1000
> >>         instances so it isn't a large deployment.
> >>
> >>         Are there any good configuration tweaks for RabbitMQ running
> >>         on OpenStack Queens?
> >>
> >>         Many Thanks,
> >>
> >>         --
> >>
> >>         Grant Morley
> >>         Cloud Lead, Civo Ltd
> >>         www.civo.com <https://www.civo.com/>| Signup for an account!
> >>         <https://www.civo.com/signup>
> >>
> >     --
> >
> >     Grant Morley
> >     Cloud Lead, Civo Ltd
> >     www.civo.com <https://www.civo.com/>| Signup for an account!
> >     <https://www.civo.com/signup>
> >
>
>

-- 
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190807/475a6bb0/attachment-0001.html>