Another thing to check if you're having seemingly inexplicable messaging
issues is that there isn't a notification queue filling up somewhere. If
notifications are enabled somewhere but nothing is consuming them the
size of the queue will eventually grind rabbit to a halt.
I used to check queue sizes through the rabbit web ui, so I have to
admit I'm not sure how to do it through the cli.
On 7/31/19 10:48 AM, Gabriele Santomaggio wrote:
> Hi,
> Are you using ssl connections ?
>
> Can be this issue ?
> https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1800957
>
>
> ------------------------------------------------------------------------
> *From:* Laurent Dumont <laurentfdumont@gmail.com>
> *Sent:* Wednesday, July 31, 2019 4:20 PM
> *To:* Grant Morley
> *Cc:* openstack-operators@lists.openstack.org
> *Subject:* Re: Slow instance launch times due to RabbitMQ
> That is a bit strange, list_queues should return stuff. Couple of ideas :
>
> * Are the Rabbit connection failure logs on the compute pointing to a
> specific controller?
> * Are there any logs within Rabbit on the controller that would point
> to a transient issue?
> * cluster_status is a snapshot of the cluster at the time you ran the
> command. If the alarms have cleared, you won't see anything.
> * If you have the RabbitMQ management plugin activated, I would
> recommend a quick look to see the historical metrics and overall status.
>
>
> On Wed, Jul 31, 2019 at 9:35 AM Grant Morley <grant@civo.com
> <mailto:grant@civo.com>> wrote:
>
> Hi guys,
>
> We are using Ubuntu 16 and OpenStack ansible to do our setup.
>
> rabbitmqctl list_queues
> Listing queues
>
> (Doesn't appear to be any queues )
>
> rabbitmqctl cluster_status
>
> Cluster status of node
> 'rabbit@management-1-rabbit-mq-container-b4d7791f'
> [{nodes,[{disc,['rabbit@management-1-rabbit-mq-container-b4d7791f',
> 'rabbit@management-2-rabbit-mq-container-b455e77d',
> 'rabbit@management-3-rabbit-mq-container-1d6ae377']}]},
> {running_nodes,['rabbit@management-3-rabbit-mq-container-1d6ae377',
> 'rabbit@management-2-rabbit-mq-container-b455e77d',
> 'rabbit@management-1-rabbit-mq-container-b4d7791f']},
> {cluster_name,<<"openstack">>},
> {partitions,[]},
> {alarms,[{'rabbit@management-3-rabbit-mq-container-1d6ae377',[]},
> {'rabbit@management-2-rabbit-mq-container-b455e77d',[]},
> {'rabbit@management-1-rabbit-mq-container-b4d7791f',[]}]}]
>
> Regards,
>
> On 31/07/2019 11:49, Laurent Dumont wrote:
>> Could you forward the output of the following commands on a
>> controller node? :
>>
>> rabbitmqctl cluster_status
>> rabbitmqctl list_queues
>>
>> You won't necessarily see a high load on a Rabbit cluster that is
>> in a bad state.
>>
>> On Wed, Jul 31, 2019 at 5:19 AM Grant Morley <grant@civo.com
>> <mailto:grant@civo.com>> wrote:
>>
>> Hi all,
>>
>> We are randomly seeing slow instance launch / deletion times
>> and it appears to be because of RabbitMQ. We are seeing a lot
>> of these messages in the logs for Nova and Neutron:
>>
>> ERROR oslo.messaging._drivers.impl_rabbit [-]
>> [f4ab3ca0-b837-4962-95ef-dfd7d60686b6] AMQP server on
>> 10.6.2.212:5671 <http://10.6.2.212:5671> is unreachable: Too
>> many heartbeats missed. Trying again in 1 seconds. Client
>> port: 37098: ConnectionForced: Too many heartbeats missed
>>
>> The RabbitMQ cluster isn't under high load and I am not seeing
>> any packets drop over the network when I do some tracing.
>>
>> We are only running 15 compute nodes currently and have >1000
>> instances so it isn't a large deployment.
>>
>> Are there any good configuration tweaks for RabbitMQ running
>> on OpenStack Queens?
>>
>> Many Thanks,
>>
>> --
>>
>> Grant Morley
>> Cloud Lead, Civo Ltd
>> www.civo.com <https://www.civo.com/>| Signup for an account!
>> <https://www.civo.com/signup>
>>
> --
>
> Grant Morley
> Cloud Lead, Civo Ltd
> www.civo.com <https://www.civo.com/>| Signup for an account!
> <https://www.civo.com/signup>
>