<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Kris,</div><div class=""><br class=""></div><div class="">You’re right this system work well before. I restarted ,this time, rabbitmq-server, nova-api, nova-conductor ,nova-console, nova-novncproxy and nova-scheduler together in a single line on Controller node. Now it’s back normal. so for lessen learned, if restarting singe service not resolve problem then think about restarting them all if related together.</div><div class=""><br class=""></div><div class="">Thank you Kris.</div><div class=""><br class=""></div><div class="">Evan.</div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 25, 2015, at 12:12 AM, Kris G. Lindgren <<a href="mailto:klindgren@godaddy.com" class="">klindgren@godaddy.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">
<div class="">Evan,</div>
<div class=""><br class="">
</div>
<div class="">I would recommend restarting anything that takes to rabbitmq. This is a case of your rpc workers thinking that they are connected to rabbitmq but they really aren't. I assume that this setup use to work at one time and now doesn’t?</div>
<div class="">
<div id="MAC_OUTLOOK_SIGNATURE" class="">
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;"><br class="">
</span></font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">___________________________________________________________________</span></font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">Kris Lindgren</span></font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">Senior Linux Systems Engineer</span></font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">GoDaddy</span></font></font></div>
</div>
</div>
</div>
</div>
<div class=""><br class="">
</div>
<span id="OLK_SRC_BODY_SECTION" class="">
<div style="font-family: Calibri; font-size: 12pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">From: </span>Evan <<a href="mailto:evandeng2009@qq.com" class="">evandeng2009@qq.com</a>><br class="">
<span style="font-weight:bold" class="">Date: </span>Tuesday, November 24, 2015 at 5:02 AM<br class="">
<span style="font-weight:bold" class="">To: </span>OpenStack-operators <<a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a>><br class="">
<span style="font-weight:bold" class="">Subject: </span>[Openstack-operators] Can't boot image getting nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID msg_id<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">hi,<br class="">
<br class="">
I keep getting such error(as subject) when trying to boot an image for instance, no matter from webpage dashboard or cli. firstly it show the status of scheduling and after a while it's Error. I checked status of all service ,nothing gets failed. Below are
the logs for nova-api and nova-conductor. <br class="">
<br class="">
It seems like the problem of rabbitmq, but it's running well without any explicit error shown. Once I tried to restart the rabbitmq then instance created successfully. Before this case happens I tried to set a 2nd external network on Network node but failed
so I rolled it back(changed configs back and restart related services). Also checked those common CLI they're all good like nova image-list, nova service-list,neutron agent-list ,keystone service-list.<br class="">
<br class="">
Openstack-Juno, with 1 controller, 1 network, 3 compute, 1 block and 1 object, servers have much more resources than needed.<br class="">
<br class="">
<br class="">
<span style="font-weight: bold;" class="">controller# grep MessagingTimeout /var/log/nova/nova-api.log</span><br class="">
......<br class="">
2015-11-24 15:58:00.404 18197 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 039995083875445bacdd3bd8667cda51<br class="">
2015-11-24 15:59:02.618 18204 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 58c3ea4f3c2d43699632c1999df79e91<br class="">
2015-11-24 18:58:28.030 18201 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 6b43bf60e2cd494fab116515c5997ab5<br class="">
<br class="">
<span style="font-weight: bold;" class="">controller# grep ERROR /var/log/nova/nova-api.log</span><br class="">
......<br class="">
2015-11-24 18:57:27.003 18201 ERROR oslo.messaging._drivers.impl_rabbit [req-2b810cc0-d809-4f2c-8a01-d1f7eca62b48 ] Failed to consume message from queue:
<span style="color: rgb(255, 0, 0);" class="">[Errno 104] Connection reset by peer</span><br class="">
2015-11-24 18:58:28.030 18201 ERROR nova.api.openstack [req-2b810cc0-d809-4f2c-8a01-d1f7eca62b48 None] Caught error: Timed out waiting for a reply to message ID 6b43bf60e2cd494fab116515c5997ab5<br class="">
2015-11-24 19:07:26.591 18200 ERROR oslo.messaging._drivers.impl_rabbit [req-f532b095-5f37-432d-8cdc-06db8498b3c1 ] Failed to publish message to topic 'conductor': [Errno 104] Connection reset by peer<br class="">
2015-11-24 19:22:36.310 18183 ERROR oslo.messaging._drivers.impl_rabbit [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ]
<span style="color: rgb(255, 0, 0);" class="">Failed to publish message to topic 'conductor': [Errno 104] Connection reset by peer</span><br class="">
<br class="">
<span style="font-weight: bold;" class=""># tail -n 80 /var/log/nova/nova-conductor.log</span><br class="">
......<br class="">
2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send<br class="">
2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)<br class="">
2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer<br class="">
2015-11-24 19:22:37.356 17770 TRACE oslo.messaging._drivers.impl_rabbit <br class="">
2015-11-24 19:22:37.357 17770 INFO oslo.messaging._drivers.impl_rabbit [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Delaying reconnect for 1.0 seconds...<br class="">
2015-11-24 19:22:38.358 17770 INFO oslo.messaging._drivers.impl_rabbit [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Connecting to AMQP server on controller:5672<br class="">
2015-11-24 19:22:38.379 17770 INFO oslo.messaging._drivers.impl_rabbit [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa ] Connected to AMQP server on controller:5672<br class="">
2015-11-24 19:23:38.385 17770 ERROR nova.scheduler.driver [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa None] Exception during scheduler.run_instance<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver Traceback (most recent call last):<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 616, in build_instances<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver request_spec, filter_properties)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 49, in select_destinations<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver context, request_spec, filter_properties)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver return getattr(self.instance, __name)(*args, **kwargs)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 34, in select_destinations<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver context, request_spec, filter_properties)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 108, in select_destinations<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver request_spec=request_spec, filter_properties=filter_properties)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver retry=self.retry)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver timeout=timeout, retry=retry)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver retry=retry)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver result = self._waiter.wait(msg_id, timeout)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver reply, ending = self._poll_connection(msg_id, timeout)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver % msg_id)<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver MessagingTimeout: Timed out waiting for a reply to message ID d015995266a24e05b21ff0cd2ff3c6bc<br class="">
2015-11-24 19:23:38.385 17770 TRACE nova.scheduler.driver <br class="">
2015-11-24 19:23:38.388 17770 WARNING nova.scheduler.driver [req-24c4ce2a-6516-401e-a6f8-8a89758cc6fa None] [instance: 66147c0d-fa53-4e2b-8447-d4f20ac7bcbb] Setting instance to ERROR state.<br class="">
<br class="">
<span style="font-weight: bold;" class="">controller# systemctl status rabbitmq-server</span><br class="">
rabbitmq-server.service - RabbitMQ broker<br class="">
Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled)<br class="">
Active: <span style="color: rgb(51, 153, 102);" class="">active (running) </span>since Tue 2015-11-24 15:02:41 CST; 4h 43min ago<br class="">
Process: 137583 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)<br class="">
Main PID: 137639 (beam.smp)<br class="">
CGroup: /system.slice/rabbitmq-server.service<br class="">
├─137639 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_s...<br class="">
├─137654 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon<br class="">
├─137767 inet_gethost 4<br class="">
└─137768 inet_gethost 4<br class="">
<br class="">
Nov 24 15:02:40 controller systemd[1]: rabbitmq-server.service: Got notification message from PID 137701, but reception only permitted for PID 137639<br class="">
Nov 24 15:02:40 controller systemd[1]: rabbitmq-server.service: Got notification message from PID 137702, but reception only permitted for PID 137639<br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: RabbitMQ 3.3.5. Copyright (C) 2007-2014 GoPivotal, Inc.<br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: ## ## Licensed under the MPL. See
<a href="http://www.rabbitmq.com/" class="">http://www.rabbitmq.com/</a><br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: ## ##<br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: ########## Logs: <a href="mailto:/var/log/rabbitmq/rabbit@controller.log" class="">
/var/log/rabbitmq/rabbit@controller.log</a><br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: ###### ## <a href="mailto:/var/log/rabbitmq/rabbit@controller-sasl.log" class="">
/var/log/rabbitmq/rabbit@controller-sasl.log</a><br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: ##########<br class="">
Nov 24 15:02:41 controller rabbitmq-server[137639]: Starting broker... completed with 6 plugins.<br class="">
Nov 24 15:02:41 controller systemd[1]: Started RabbitMQ broker.<br class="">
</div>
</div>
</span>
</div>
</div></blockquote></div><br class=""></body></html>