Re: [ops] rabbitmq queues for nova versioned notifications queues keep filling up
Thank you, Christopher. From: cmccarth@mathworks.com At: 06/04/21 11:17:23 UTC-4:00To: openstack-discuss@lists.openstack.org Subject: Re: [ops] rabbitmq queues for nova versioned notifications queues keep filling up Hi Ajay, We work around this by setting a TTL on our notifications queues via RabbitMQ policy definition. We include the following in our definitions.json for RabbitMQ: "policies":[ {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600000}, "priority":0} ] This expires messages in the notifications and versioned_notifications queues after 10 minutes, which seems to work well for us. I believe we initially picked up this workaround from this[1] bug report. Hope this helps, - Chris -- Christopher McCarthy MathWorks cmccarth@mathworks.com [1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1737170 Date: Wed, 2 Jun 2021 22:39:54 -0000 From: "Ajay Tikoo (BLOOMBERG/ 120 PARK)" <atikoo@bloomberg.net> To: openstack-discuss@lists.openstack.org Subject: [ops] rabbitmq queues for nova versioned notifications queues keep filling up Message-ID: <60B808BA00D0068401D80001_0_3025859@msclnypmsgsv04> Content-Type: text/plain; charset="utf-8" I am not sure if this is the right channel/format to post this question, so my apologies in advance if this is not the right place. We are using Openstack Rocky. Watcher needs versioned notifications to be enabled. However after enabling versioned notifications, the queues for versioned_notifications (info and error) keep filling up Based on the updates the the Watchers cluster data model, it appears that Watcher is consuming messages, but they still linger in these queues. So with nova versioned notifications disabled, Watcher is unable to update the cluster data model (between rebuild intervals), and with them enabled, it keeps filling up the MQ queues. What is the best way to resolve this? Thank you, Ajay Tikoo
On Mon, 2021-06-07 at 13:56 +0000, Ajay Tikoo (BLOOMBERG/ 120 PARK) wrote:
Thank you, Christopher.
From: cmccarth@mathworks.com At: 06/04/21 11:17:23 UTC-4:00To: openstack-discuss@lists.openstack.org Subject: Re: [ops] rabbitmq queues for nova versioned notifications queues keep filling up
Hi Ajay,
We work around this by setting a TTL on our notifications queues via RabbitMQ policy definition. We include the following in our definitions.json for RabbitMQ:
"policies":[ {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600000}, "priority":0} ]
adding the oslo and nova tag as im wonderign if ^ should be configurable via oslo.messagaing or nova automatically. perhaps we have a configuration option fo notification experation but i think this could be a useful feature to add if its not already present. we have a rabbit_transient_queues_ttl https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messag... but im not sure tha tthat is applied to notification queues by default. im wondering if that is a bug that we shoudl correct? the default value is 1800 sec which is 30 mins which seams reasonable. while longer then the 10 mins chris is usign it's better then infinity.
This expires messages in the notifications and versioned_notifications queues after 10 minutes, which seems to work well for us. I believe we initially picked up this workaround from this[1] bug report.
Hope this helps,
- Chris
-- Christopher McCarthy MathWorks cmccarth@mathworks.com
[1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1737170
Date: Wed, 2 Jun 2021 22:39:54 -0000 From: "Ajay Tikoo (BLOOMBERG/ 120 PARK)" <atikoo@bloomberg.net> To: openstack-discuss@lists.openstack.org Subject: [ops] rabbitmq queues for nova versioned notifications queues keep filling up Message-ID: <60B808BA00D0068401D80001_0_3025859@msclnypmsgsv04> Content-Type: text/plain; charset="utf-8"
I am not sure if this is the right channel/format to post this question, so my apologies in advance if this is not the right place.
We are using Openstack Rocky. Watcher needs versioned notifications to be enabled. However after enabling versioned notifications, the queues for versioned_notifications (info and error) keep filling up Based on the updates the the Watchers cluster data model, it appears that Watcher is consuming messages, but they still linger in these queues. So with nova versioned notifications disabled, Watcher is unable to update the cluster data model (between rebuild intervals), and with them enabled, it keeps filling up the MQ queues. What is the best way to resolve this?
Thank you, Ajay Tikoo
participants (2)
-
Ajay Tikoo (BLOOMBERG/ 120 PARK)
-
Sean Mooney