Open Stack

We see the same behavior, and a lot of other people have this problem, too.  It's related to how the Rabbit client code is handling disconnects/reconnects.  Do you happen to be connecting to Rabbit through a load balancer?

We don't have a create solution for this, but talked to several folks at the Summit for ideas.

See this bug:  https://bugs.launchpad.net/oslo/+bug/856764  and this review: https://review.openstack.org/#/c/76686/ for a possible solution.  This has not been merged in yet, though.

Mike


From: Belmiro Moreira <moreira.belmiro.email.lists at gmail.com<mailto:moreira.belmiro.email.lists at gmail.com>>
Date: Wednesday, May 21, 2014 at 8:05 AM
To: OpenStack Operators <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: [Openstack-operators] nova-compute not consuming messages

Hi,
In our infrastructure we are observing that some compute nodes are reporting the state correctly but not consuming messages.
For the scheduler they are available, so it schedules VMs on them, and then the messages are pilling up in the compute."host" queues.
Restarting nova-compute solves the problem. All old messages in the queue are consumed and they start behaving properly until next time...
I don't get anything interesting from the logs.

We are using rabbit 3.2.4 clustered with mirror queues and Havana 2.

Anyone observed the same symptoms?

thanks,
Belmiro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140521/de73147f/attachment.html>