<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div><br>
</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div>Hi</div>
<div>I am seeing on Kilo if I bring down one contoller node sometimes some computes report down forever.</div>
<div>I need to restart the compute service on compute node to recover. Looks like oslo is not reconnecting in nova-compute</div>
<div>Here is the Trace from nova-compute</div>
<div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db retry=self.retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db timeout=timeout, retry=retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db retry=retry)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db result = self._waiter.wait(msg_id, timeout)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db message = self.waiters.get(msg_id, timeout=timeout)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db 'to message ID %s' % msg_id)</div>
<div>2016-04-19 20:25:39.090 6 TRACE nova.servicegroup.drivers.db MessagingTimeout: Timed out waiting for a reply to message ID e064b5f6c8244818afdc5e91fff8ebf1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Any thougths. I am at stable/kilo for oslo</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
</body>
</html>