<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>See something similar with heartbeat seems like reconnection attempt fails</div>
<div>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:27:01.294 6 DEBUG nova.openstack.common.loopingcall [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Dynamic looping call <bound method Service.periodic_tasks of <nova.service.Service object at 0x46a1c50>> sleeping for 60.00 seconds _inner
/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:132</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:01.294 6 DEBUG nova.openstack.common.periodic_task [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:01.295 6 DEBUG nova.openstack.common.periodic_task [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:01.295 6 DEBUG nova.openstack.common.periodic_task [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:01.295 6 DEBUG nova.openstack.common.periodic_task [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Running periodic task ComputeManager._cleanup_running_deleted_instances run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.421 6 ERROR oslo_messaging._drivers.impl_rabbit [-] Declaring queue failed with (Socket closed), retrying</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: Socket closed</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver Traceback (most recent call last):</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 228, in poll</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver self.conn.consume(limit=1)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1194, in consume</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver six.next(it)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1105, in iterconsume</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver error_callback=_error_callback)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 885, in ensure</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver 'retry': retry}</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver TypeError: %d format: a number is required, not NoneType</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.422 6 TRACE oslo_messaging._drivers.amqpdriver</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:28:48.430 6 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 10.23.221.110:5672 is unreachable: connection already closed. Trying again in 1 seconds.</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 ERROR nova.openstack.common.periodic_task [req-9c9785ed-2598-4b95-a40c-307f8d7e8416 - - - - -] Error during ComputeManager._cleanup_running_deleted_instances: Timed out waiting for a reply to message ID c0c46bd3ebfb4441981617e089c5a18d</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task task(self, context)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6410, in _cleanup_running_deleted_instances</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task for instance in self._running_deleted_instances(context):</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6464, in _running_deleted_instances</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task instances = self._get_instances_on_driver(context, filters)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 796, in _get_instances_on_driver</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task context, filters, use_slave=True)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task args, kwargs)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task objver=objver, args=args, kwargs=kwargs)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task retry=self.retry)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task timeout=timeout, retry=retry)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 381, in send</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task retry=retry)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 370, in _send</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task result = self._waiter.wait(msg_id, timeout)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 274, in wait</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task message = self.waiters.get(msg_id, timeout=timeout)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 180, in get</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task 'to message ID %s' % msg_id)</p>
<p style="margin: 0px; font-size: 12px; line-height: normal; font-family: Menlo;">
2016-04-21 15:29:01.302 6 TRACE nova.openstack.common.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID c0c46bd3ebfb4441981617e089c5a18d</p>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 12:11 PM<br>
<span style="font-weight:bold">To: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Thanks Kris that’s good information will try out your suggestions</div>
<div>Ajay</div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 12:08 PM<br>
<span style="font-weight:bold">To: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>We just use heartbeat. But from what I recall other people have good luck with both set. I would keep them if they are already set , maybe just dial down how aggressive they are. One thing I should mention is that if you have a large number of RPC workers,
enabling heartbeats will increase cpu consumption about 1-2% per worker (in our experience). Since its now doing something with rabbitmq every few seconds. This can also increase load on the rabbitmq side as well. For us having a stable rabbit environment
is well worth the tradeoff.</div>
<div>
<div id="">
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><br>
</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">___________________________________________________________________</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">Kris Lindgren</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">Senior Linux Systems Engineer</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">GoDaddy</span></font></font></div>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Ajay Kalambur (akalambu)" <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 1:04 PM<br>
<span style="font-weight:bold">To: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Do you recommend both or can I do away with the system timers and just keep the heartbeat?</div>
<div>Ajay</div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 11:54 AM<br>
<span style="font-weight:bold">To: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Yea, that only fixes part of the issue. The other part is getting the openstack messaging code itself to figure out the connection its using is no longer valid. Heartbeats by itself solved 90%+ of our issues with rabbitmq and nodes being disconnected
and never reconnecting.</div>
<div>
<div id="">
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><br>
</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">___________________________________________________________________</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">Kris Lindgren</span></font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">Senior Linux Systems Engineer</span></font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">GoDaddy</span></font></font></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Ajay Kalambur (akalambu)" <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 12:51 PM<br>
<span style="font-weight:bold">To: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Trying that now. I had aggressive system keepalive timers before</div>
<div><br>
</div>
<div>
<div>net.ipv4.tcp_keepalive_intvl = 10</div>
<div>net.ipv4.tcp_keepalive_probes = 9</div>
<div>net.ipv4.tcp_keepalive_time = 5</div>
</div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Kris G. Lindgren" <<a href="mailto:klindgren@godaddy.com">klindgren@godaddy.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 11:50 AM<br>
<span style="font-weight:bold">To: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>>, "<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Do you have rabbitmq/oslo messaging heartbeats enabled?</div>
<div><br>
</div>
<div>If you aren't using heartbeats it will take a long time for the nova-compute agent to figure out that its actually no longer attached to anything. Heartbeat does periodic checks against rabbitmq and will catch this state and reconnect. </div>
<div>
<div id="">
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><br>
</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">___________________________________________________________________</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri">Kris Lindgren</font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">Senior Linux Systems Engineer</span></font></font></div>
<div><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" face="Calibri"><span class="Apple-style-span" style="font-size: 14px;">GoDaddy</span></font></font></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"Ajay Kalambur (akalambu)" <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 at 11:43 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Openstack-operators] [oslo]nova compute reconnection Issue Kilo<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
</div>
<div>Hi</div>
<div>I am seeing on Kilo if I bring down one contoller node sometimes some computes report down forever.</div>
<div>I need to restart the compute service on compute node to recover. Looks like oslo is not reconnecting in nova-compute</div>
<div>Here is the Trace from nova-compute</div>
<div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db retry=self.retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db timeout=timeout, retry=retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db retry=retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db result = self._waiter.wait(msg_id, timeout)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db message = self.waiters.get(msg_id, timeout=timeout)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db 'to message ID %s' % msg_id)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db MessagingTimeout: Timed out waiting for a reply to message ID e064b5f6c8244818afdc5e91fff8ebf1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Any thougths. I am at stable/kilo for oslo</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
</div>
</div>
</span></div>
</div>
</span></div>
</div>
</span></div>
</div>
</span></div>
</div>
</span></div>
</div>
</span></div>
</div>
</span>
</body>
</html>