Open Stack

Yeah, we've experienced it but hadn't had time yet to really dig in like this, or gotten a good workaround. If you file a bug, please let me know what number.

Thanks,
Kevin
________________________________________
From: Sam Morrison [sorrison at gmail.com]
Sent: Sunday, July 24, 2016 11:27 PM
To: OpenStack Operators
Subject: [Openstack-operators] [oslo] RabbitMQ queue TTL issues moving to       Liberty

We recently upgraded to Liberty and have come across some issues with queue build ups.

This is due to changes in rabbit to set queue expiries as opposed to queue auto delete.
See https://bugs.launchpad.net/oslo.messaging/+bug/1515278 for more information.

The fix for this bug is in liberty and it does fix an issue however it causes another one.

Every time you restart something that has a fanout queue. Eg. cinder-scheduler or the neutron agents you will have
a queue in rabbit that is still bound to the rabbitmq exchange (and so still getting messages in) but no consumers.

These messages in these queues are basically rubbish and don’t need to exist. Rabbit will delete these queues after 10 mins (although the default in master is now changed to 30 mins)

During this time the queue will grow and grow with messages. This sets off our nagios alerts and our ops guys have to deal with something that isn’t really an issue. They basically delete the queue.

A bad scenario is when you make a change to your cloud that means all your 1000 neutron agents are restarted, this causes a couple of dead queues per agent to hang around. (port updates and security group updates) We get around 25 messages / second on these queues and so you can see after 10 minutes we have a ton of messages in these queues.

1000 x 2 x 25 x 600 = 30,000,000 messages in 10 minutes to be precise.

Has anyone else been suffering with this before a raise a bug?

Cheers,
Sam


_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators