Open Stack

Hi Ajay,

We work around this by setting a TTL on our notifications queues via RabbitMQ policy definition. We include the following in our definitions.json for RabbitMQ:

"policies":[
    {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600000}, "priority":0}  ]

This expires messages in the notifications and versioned_notifications queues after 10 minutes, which seems to work well for us. I believe we initially picked up this workaround from this[1] bug report.

Hope this helps,

- Chris

--
Christopher McCarthy
MathWorks
cmccarth at mathworks.com

[1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1737170


Date: Wed, 2 Jun 2021 22:39:54 -0000
From: "Ajay Tikoo (BLOOMBERG/ 120 PARK)" <atikoo at bloomberg.net>
To: openstack-discuss at lists.openstack.org
Subject: [ops] rabbitmq queues for nova versioned notifications queues
keep filling up
Message-ID: <60B808BA00D0068401D80001_0_3025859 at msclnypmsgsv04>
Content-Type: text/plain; charset="utf-8"

I am not sure if this is the right channel/format to post this question, so my apologies in advance if this is not the right place.

We are using Openstack Rocky. Watcher needs versioned notifications to be enabled. However after enabling versioned notifications, the queues for versioned_notifications (info and error) keep filling up Based on the updates the the Watchers cluster data model, it appears that Watcher is consuming messages, but they still linger in these queues. So with nova versioned notifications disabled, Watcher is unable to update the cluster data model (between rebuild intervals), and with them enabled, it keeps filling up the MQ queues. What is the best way to resolve this?

Thank you,
Ajay Tikoo

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210604/ec18896d/attachment.html>