<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Hi</div>
<div>We see an issue where if we boot 10 instances on 1 compute node at a time we see that sometimes a few instances error out on compute node with following Traceback</div>
<div>It looks like the RPC message call from compute saying set Instance to Active is timing out. It seems to help to set rpc response timeout to 180 seconds. But what we see from TCP dump is that the message reaches the conductor and conductor replies almost
instantly. It looks like the compute node in question somehow takes a long time to process it. Any pointers. Also we had to increase heartbeat timeout to 120 from 60 to make this scenario work better. Any inputs on what maybe going on here?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:44:19.518 6 INFO nova.virt.libvirt.driver [-] [instance: 31e67591-6bd4-4df9-bba1-7a1d6f785be0] Instance spawned successfully.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:44:55.692 6 INFO nova.compute.manager [req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 31e67591-6bd4-4df9-bba1-7a1d6f785be0] During sync_power_state the instance has a pending
task (spawning). Skip.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:44:55.693 6 INFO nova.compute.manager [req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 31e67591-6bd4-4df9-bba1-7a1d6f785be0] VM Started (Lifecycle Event)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:45:04.544 6 INFO nova.virt.libvirt.driver [-] [instance: 5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] Instance spawned successfully.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:45:04.558 6 INFO nova.virt.libvirt.driver [-] [instance: ba4b3c89-0f57-4c46-91dd-44434012d289] Instance spawned successfully.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:45:04.607 6 INFO nova.compute.manager [req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] VM Resumed (Lifecycle Event)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:45:16.669 6 INFO nova.virt.libvirt.driver [-] [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] Instance spawned successfully.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:45:16.710 6 INFO nova.compute.manager [req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] VM Started (Lifecycle Event)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [req-20ec8e07-d57c-4679-8122-4cfa990b6d6f 168b323822284084b1c1393faeb5b9e1 aa39823c34e4496793250c0bc5cf7a31 - - -] [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c]
Unexpected build failure, not rescheduling build.</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] Traceback (most recent call last):</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] filter_properties)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2082, in _build_and_run_instance</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] instance.save(expected_task_state=task_states.SPAWNING)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in wrapper</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] ctxt, self, fn.__name__, args, kwargs)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in object_action</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] objmethod=objmethod, args=args, kwargs=kwargs)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] retry=self.retry)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] timeout=timeout, retry=retry)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] retry=retry)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] result = self._waiter.wait(msg_id, timeout)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] message = self.waiters.get(msg_id, timeout=timeout)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] 'to message ID %s' % msg_id)</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c] MessagingTimeout: Timed out waiting for a reply to message ID 7cb74c7df8e148e9bd3d3cee2e433b70</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<br>
</p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures">2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 3794b761-f623-4fb9-a91a-f2dffca42c4c]</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures"><br>
</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures"><br>
</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">
<span style="font-variant-ligatures: no-common-ligatures"><br>
</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(255, 254, 254);">Ajay</span></p>
<p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;">
<span style="font-variant-ligatures: no-common-ligatures; background-color: rgb(255, 254, 254);"><br>
</span></p>
</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
</body>
</html>