Open Stack

Hi,

I’ve been chasing a bug in ironic’s neutron agent for the last few days and
I think its time to ask for some advice.

Specifically, I was asked to debug why a set of controllers was using so
much RAM, and the answer was that rabbitmq had a queue called
ironic-neutron-agent-heartbeat.info with 800,000 messages enqueued. This
notification queue is used by ironic’s neutron agent to calculate the hash
ring. I have been able to duplicate this issue in a stock kolla-ansible
install with ironic turned on but no bare metal nodes enrolled in ironic.
About 0.6 messages are queued per second.

I added some debugging code (hence the thread yesterday about mangling the
code kolla deploys), and I can see that the messages in the queue are being
read by the ironic neutron agent and acked correctly. However, they are not
removed from the queue.

You can see your queue size while using kolla with this command:

docker exec rabbitmq rabbitmqctl list_queues messages name messages_ready
consumers  | sort -n | tail -1

My stock install that’s been running for about 12 hours currently has 8,244
messages in that queue.

Where I’m a bit stumped is I had assumed that the messages weren’t being
acked correctly, which is not the case. Is there something obvious about
notification queues like them being persistent that I’ve missed in my
general ignorance of the underlying implementation of notifications?

Thanks,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190205/106f0a22/attachment.html>