Open Stack

Thanks a lot Jay! Exactly what I was looking for. Will take a look at the
bug.

My colleague fixed it by creating a periodic task which forces the queues
to be declared. But we are not sure if there are consequences other than it
being not so nice.

Sam


On Thu, Aug 29, 2013 at 2:12 PM, Jay Pipes <jaypipes at gmail.com> wrote:

> On 08/29/2013 03:50 AM, Sam Stoelinga wrote:
>
>> Hi,
>>
>> We have the following setup:
>> Folsom version
>> 1 controller running all apis and also nova-compute
>> 1 compute node only running nova-compute and nova-network
>>
>> Problem is that when we reboot the controller the compute node
>> reconnects successfully according to the nova-compute and nova-network
>> logs, but it won't accept any messages from rabbitmq. Seems to be that
>> the consumer didn't actually reconnect.
>>
>> I noticed that in impl_kombu.py:Connection._**connect on line 457 the
>> variable self.consumers is an empty list. I'm not sure if this is normal.
>>
>> When launching an instance the compute node never receives the message
>> from rabbitmq. If we use durable queues the message will stay in the
>> queue until nova-compute is manually restarted. And if we don't use
>> durable queues the queue for the compute node doesn't get
>> declared/created at all.
>>
>> Another side note is that if we only stop rabbitmq server for 5 minutes
>> and later start it again. Everything works fine. So we only have this
>> problem when executing the command: $ reboot on the controller.
>>
>> The logs look like it's normal, but in fact it's not see nova-compute.log:
>> 2013-08-29 15:38:49 27818 INFO nova.openstack.common.rpc.**common [-]
>> Reconnecting to AMQP server on 172.16.200.1:5672 <
>> http://172.16.200.1:5672>
>>
>> 2013-08-29 15:38:49 27818 ERROR nova.openstack.common.rpc.**common [-]
>> AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672> is
>>
>> unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
>> 2013-08-29 15:39:03 27818 ERROR nova.service [-] Recovered model server
>> connection!
>> 2013-08-29 15:39:03 27818 DEBUG nova.service [-] heart_beat rpc send...
>> report_state /usr/lib/python2.7/dist-**packages/nova/service.py:558
>> 2013-08-29 15:39:13 27818 DEBUG nova.service [-] heart_beat rpc send...
>> report_state /usr/lib/python2.7/dist-**packages/nova/service.py:558
>> 2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.**common [-]
>> Reconnecting to AMQP server on 172.16.200.1:5672 <
>> http://172.16.200.1:5672>
>>
>> 2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.**common [-]
>> Connected to AMQP server on 172.16.200.1:5672 <http://172.16.200.1:5672>
>>
>>
>> Anybody any suggestions on where to look or any hot fixes that can be
>> applied?
>>
>
> It's this bug, more than likely:
>
> https://bugs.launchpad.net/**oslo/+bug/856764<https://bugs.launchpad.net/oslo/+bug/856764>
>
> It's a major issue, IMO. My colleague Kevin Bringard has a patch (see the
> bug) that fixes this issue. We're currently manually patching our OpenStack
> services with this fix until we can get the fix into oslo.rpc. Note,
> however, that the fix depends on a newer version of kombu than is in Ubuntu
> 12.04 LTS at this time...
>
> Read the discussion on the bug for more information.
>
> Best,
> -jay
>
>
> ______________________________**_________________
> Mailing list: http://lists.openstack.org/**cgi-bin/mailman/listinfo/**
> openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/**cgi-bin/mailman/listinfo/**
> openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130830/81c357d3/attachment.html>