<div dir="ltr">Thanks a lot Jay! Exactly what I was looking for. Will take a look at the bug.<div><br></div><div>My colleague fixed it by creating a periodic task which forces the queues to be declared. But we are not sure if there are consequences other than it being not so nice.<div>
<br></div><div>Sam</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 29, 2013 at 2:12 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 08/29/2013 03:50 AM, Sam Stoelinga wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Hi,<br>
<br>
We have the following setup:<br>
Folsom version<br>
1 controller running all apis and also nova-compute<br>
1 compute node only running nova-compute and nova-network<br>
<br>
Problem is that when we reboot the controller the compute node<br>
reconnects successfully according to the nova-compute and nova-network<br>
logs, but it won't accept any messages from rabbitmq. Seems to be that<br>
the consumer didn't actually reconnect.<br>
<br>
I noticed that in impl_kombu.py:Connection._<u></u>connect on line 457 the<br>
variable self.consumers is an empty list. I'm not sure if this is normal.<br>
<br>
When launching an instance the compute node never receives the message<br>
from rabbitmq. If we use durable queues the message will stay in the<br>
queue until nova-compute is manually restarted. And if we don't use<br>
durable queues the queue for the compute node doesn't get<br>
declared/created at all.<br>
<br>
Another side note is that if we only stop rabbitmq server for 5 minutes<br>
and later start it again. Everything works fine. So we only have this<br>
problem when executing the command: $ reboot on the controller.<br>
<br>
The logs look like it's normal, but in fact it's not see nova-compute.log:<br>
2013-08-29 15:38:49 27818 INFO nova.openstack.common.rpc.<u></u>common [-]<br></div>
Reconnecting to AMQP server on <a href="http://172.16.200.1:5672" target="_blank">172.16.200.1:5672</a> <<a href="http://172.16.200.1:5672" target="_blank">http://172.16.200.1:5672</a>><div class="im"><br>
2013-08-29 15:38:49 27818 ERROR nova.openstack.common.rpc.<u></u>common [-]<br></div>
AMQP server on <a href="http://172.16.200.1:5672" target="_blank">172.16.200.1:5672</a> <<a href="http://172.16.200.1:5672" target="_blank">http://172.16.200.1:5672</a>> is<div class="im"><br>
unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.<br>
2013-08-29 15:39:03 27818 ERROR nova.service [-] Recovered model server<br>
connection!<br>
2013-08-29 15:39:03 27818 DEBUG nova.service [-] heart_beat rpc send...<br>
report_state /usr/lib/python2.7/dist-<u></u>packages/nova/service.py:558<br>
2013-08-29 15:39:13 27818 DEBUG nova.service [-] heart_beat rpc send...<br>
report_state /usr/lib/python2.7/dist-<u></u>packages/nova/service.py:558<br>
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.<u></u>common [-]<br></div>
Reconnecting to AMQP server on <a href="http://172.16.200.1:5672" target="_blank">172.16.200.1:5672</a> <<a href="http://172.16.200.1:5672" target="_blank">http://172.16.200.1:5672</a>><div class="im"><br>
2013-08-29 15:39:19 27818 INFO nova.openstack.common.rpc.<u></u>common [-]<br></div>
Connected to AMQP server on <a href="http://172.16.200.1:5672" target="_blank">172.16.200.1:5672</a> <<a href="http://172.16.200.1:5672" target="_blank">http://172.16.200.1:5672</a>><div class="im"><br>
<br>
Anybody any suggestions on where to look or any hot fixes that can be<br>
applied?<br>
</div></blockquote>
<br>
It's this bug, more than likely:<br>
<br>
<a href="https://bugs.launchpad.net/oslo/+bug/856764" target="_blank">https://bugs.launchpad.net/<u></u>oslo/+bug/856764</a><br>
<br>
It's a major issue, IMO. My colleague Kevin Bringard has a patch (see the bug) that fixes this issue. We're currently manually patching our OpenStack services with this fix until we can get the fix into oslo.rpc. Note, however, that the fix depends on a newer version of kombu than is in Ubuntu 12.04 LTS at this time...<br>
<br>
Read the discussion on the bug for more information.<br>
<br>
Best,<br>
-jay<br>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
</blockquote></div><br></div>