Open Stack

Hi,

We have the following setup:
Folsom version
1 controller running all apis and also nova-compute
1 compute node only running nova-compute and nova-network

Problem is that when we reboot the controller the compute node reconnects
successfully according to the nova-compute and nova-network logs, but it
won't accept any messages from rabbitmq. Seems to be that the consumer
didn't actually reconnect.

I noticed that in impl_kombu.py:Connection._connect on line 457 the
variable self.consumers is an empty list. I'm not sure if this is normal.

When launching an instance the compute node never receives the message from
rabbitmq. If we use durable queues the message will stay in the queue until
nova-compute is manually restarted. And if we don't use durable queues the
queue for the compute node doesn't get declared/created at all.

Another side note is that if we only stop rabbitmq server for 5 minutes and
later start it again. Everything works fine. So we only have this problem
when executing the command: $ reboot on the controller.

The logs look like it's normal, but in fact it's not see nova-compute.log:
2013-08-29 15:38:49 27818 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 172.16.200.1:5672
2013-08-29 15:38:49 27818 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 172.16.200.1:5672 is unreachable: [Errno 111] ECONNREFUSED.
Trying again in 30 seconds.
2013-08-29 15:39:03 27818 ERROR nova.service [-] Recovered model server
connection!
2013-08-29 15:39:03 27818 DEBUG nova.service [-] heart_beat rpc send...
report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
2013-08-29 15:39:13 27818 DEBUG nova.service [-] heart_beat rpc send...
report_state /usr/lib/python2.7/dist-packages/nova/service.py:558
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 172.16.200.1:5672
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.common [-]
Connected to AMQP server on 172.16.200.1:5672

Anybody any suggestions on where to look or any hot fixes that can be
applied?


Regards,
Sam Stoelinga
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130829/774d59b0/attachment.html>