Open Stack

Following https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes, I've tagged https://bugs.launchpad.net/oslo/+bug/856764 with havana-backport-potential

From: Tim Bell
Sent: 21 May 2014 17:39
To: 'Michael Dorman'; Belmiro Moreira; OpenStack Operators
Subject: RE: [Openstack-operators] nova-compute not consuming messages


Looks like a good stable backport candidate

Tim

From: Michael Dorman [mailto:mdorman at godaddy.com]
Sent: 21 May 2014 17:22
To: Belmiro Moreira; OpenStack Operators
Subject: Re: [Openstack-operators] nova-compute not consuming messages

Actually, I lied.  This change has been merged and is part of Icehouse.

We are still running on Havana, though, and haven't actually tried it.  People we talked to have had success with it, though.


From: Michael Dorman <mdorman at godaddy.com<mailto:mdorman at godaddy.com>>
Date: Wednesday, May 21, 2014 at 9:14 AM
To: Belmiro Moreira <moreira.belmiro.email.lists at gmail.com<mailto:moreira.belmiro.email.lists at gmail.com>>, OpenStack Operators <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: Re: [Openstack-operators] nova-compute not consuming messages

We see the same behavior, and a lot of other people have this problem, too.  It's related to how the Rabbit client code is handling disconnects/reconnects.  Do you happen to be connecting to Rabbit through a load balancer?

We don't have a create solution for this, but talked to several folks at the Summit for ideas.

See this bug:  https://bugs.launchpad.net/oslo/+bug/856764  and this review: https://review.openstack.org/#/c/76686/ for a possible solution.  This has not been merged in yet, though.

Mike


From: Belmiro Moreira <moreira.belmiro.email.lists at gmail.com<mailto:moreira.belmiro.email.lists at gmail.com>>
Date: Wednesday, May 21, 2014 at 8:05 AM
To: OpenStack Operators <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: [Openstack-operators] nova-compute not consuming messages

Hi,
In our infrastructure we are observing that some compute nodes are reporting the state correctly but not consuming messages.
For the scheduler they are available, so it schedules VMs on them, and then the messages are pilling up in the compute."host" queues.
Restarting nova-compute solves the problem. All old messages in the queue are consumed and they start behaving properly until next time...
I don't get anything interesting from the logs.

We are using rabbit 3.2.4 clustered with mirror queues and Havana 2.

Anyone observed the same symptoms?

thanks,
Belmiro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140521/50782e32/attachment.html>