Open Stack

On Mon, 2021-06-07 at 13:56 +0000, Ajay Tikoo (BLOOMBERG/ 120 PARK) wrote:
> Thank you, Christopher.
> 
> From: cmccarth at mathworks.com At: 06/04/21 11:17:23 UTC-4:00To:  openstack-discuss at lists.openstack.org
> Subject: Re: [ops] rabbitmq queues for nova versioned notifications queues keep filling up
> 
>       
> 
> Hi Ajay, 
>   
> We work around this by setting a TTL on our notifications queues via RabbitMQ policy definition. We include the following in our definitions.json for RabbitMQ: 
>   
> "policies":[ 
>     {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600000}, "priority":0}  ] 
>   
adding the oslo and nova tag as im wonderign if ^ should be configurable via oslo.messagaing or nova automatically.
perhaps we have a configuration option fo notification experation but i think this could be a useful feature to add if its not already present.

we have a
 rabbit_transient_queues_ttl https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.rabbit_transient_queues_ttl
but im not sure tha tthat is applied to notification queues by default.
im wondering if that is a bug that we shoudl correct?

the default value is 1800 sec which is 30 mins which seams reasonable.
while longer then the 10 mins chris is usign it's better then infinity.





> This expires messages in the notifications and versioned_notifications queues after 10 minutes, which seems to work well for us. I believe we initially picked up this workaround from this[1] bug report. 
>   
> Hope this helps, 
>   
> - Chris 
>   
> --  
> Christopher McCarthy 
> MathWorks 
> cmccarth at mathworks.com 
>   
> [1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1737170 
>   
>   
> Date: Wed, 2 Jun 2021 22:39:54 -0000
> From: "Ajay Tikoo (BLOOMBERG/ 120 PARK)" <atikoo at bloomberg.net>
> To: openstack-discuss at lists.openstack.org
> Subject: [ops] rabbitmq queues for nova versioned notifications queues
> keep filling up
> Message-ID: <60B808BA00D0068401D80001_0_3025859 at msclnypmsgsv04>
> Content-Type: text/plain; charset="utf-8"
> 
> I am not sure if this is the right channel/format to post this question, so my apologies in advance if this is not the right place.
> 
> We are using Openstack Rocky. Watcher needs versioned notifications to be enabled. However after enabling versioned notifications, the queues for versioned_notifications (info and error) keep filling up Based on the updates the the Watchers cluster data model,  it appears that Watcher is consuming messages, but they still linger in these queues. So with nova versioned notifications disabled, Watcher is unable to update the cluster data model (between rebuild intervals), and with them enabled, it keeps filling up  the MQ queues. What is the best way to resolve this?
> 
> Thank you,
> Ajay Tikoo
> 
>   
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210608/26ef870f/attachment-0001.html>