Open Stack

On 08/29/2013 03:50 AM, Sam Stoelinga wrote:
> Hi,
>
> We have the following setup:
> Folsom version
> 1 controller running all apis and also nova-compute
> 1 compute node only running nova-compute and nova-network
>
> Problem is that when we reboot the controller the compute node
> reconnects successfully according to the nova-compute and nova-network
> logs, but it won't accept any messages from rabbitmq. Seems to be that
> the consumer didn't actually reconnect.
>
> I noticed that in impl_kombu.py:Connection._connect on line 457 the
> variable self.consumers is an empty list. I'm not sure if this is normal.
>
> When launching an instance the compute node never receives the message
> from rabbitmq. If we use durable queues the message will stay in the
> queue until nova-compute is manually restarted. And if we don't use
> durable queues the queue for the compute node doesn't get
> declared/created at all.
>
> Another side note is that if we only stop rabbitmq server for 5 minutes
> and later start it again. Everything works fine. So we only have this
> problem when executing the command: $ reboot on the controller.
>
> The logs look like it's normal, but in fact it's not see nova-compute.log:
> 2013-08-29 15:38:49 27818 INFO nova.openstack.common.rpc.common [-]
> Reconnecting to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
> 2013-08-29 15:38:49 27818 ERROR nova.openstack.common.rpc.common [-]
> AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672> is
> unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
> 2013-08-29 15:39:03 27818 ERROR nova.service [-] Recovered model server
> connection!
> 2013-08-29 15:39:03 27818 DEBUG nova.service [-] heart_beat rpc send...
> report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
> 2013-08-29 15:39:13 27818 DEBUG nova.service [-] heart_beat rpc send...
> report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
> 2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
> Reconnecting to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
> 2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
> Connected to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
>
> Anybody any suggestions on where to look or any hot fixes that can be
> applied?

It's this bug, more than likely:

https://bugs.launchpad.net/oslo/+bug/856764

It's a major issue, IMO. My colleague Kevin Bringard has a patch (see 
the bug) that fixes this issue. We're currently manually patching our 
OpenStack services with this fix until we can get the fix into oslo.rpc. 
Note, however, that the fix depends on a newer version of kombu than is 
in Ubuntu 12.04 LTS at this time...

Read the discussion on the bug for more information.

Best,
-jay