<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Vish, Jay,
<div><br>
</div>
<div>OK, this looks promising. A couple of questions…</div>
<div><br>
</div>
<div>I'm seeing this RPC timeout on the Essex 2012.1 packages released with Ubuntu 12.04. I'm assuming these packages are affected by this bug?</div>
<div><br>
</div>
<div>Why would something this fundamental not show up during Essex RC.X testing?</div>
<div><br>
</div>
<div>How best to 'fix' this for our production environment (thank god it's only the research organization!)</div>
<div><br>
</div>
<div>My previous testing of Essex (running on Diablo) didn't exhibit this problem. However during testing, I was configured using FlatDHCP networking versus our production cloud using VLAN networking. This was what lead me to believe that it might be a network
configuration issue. Apparently not.</div>
<div><br>
</div>
<div>As it stands right now, we're dead in the water, so I hope some easy fix for the Ubuntu Essex release is possible.</div>
<div><br>
</div>
<div>Thanks for everyone looking this. Hope to hear a resolution soon.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Ross</div>
<div><br>
<div>
<div>On Jun 19, 2012, at 2:13 PM, Vishvananda Ishaya wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Sorry, paste fail on the last message.
<div><br>
</div>
<div>This seems like a likely culprit:</div>
<div><br>
</div>
<div><a href="https://review.openstack.org/#/c/8339/">https://review.openstack.org/#/c/8339/</a></div>
<div><br>
</div>
<div>I'm guessing it only happens on concurrent builds? We probably need a synchronized somewhere.</div>
<div><br>
</div>
<div>Vish</div>
<div><br>
<div>
<div>
<div>On Jun 19, 2012, at 12:03 PM, Jay Pipes wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div>cc'ing Vish on this, as this is now occurring on every single devstack + Tempest run, for multiple servers.<br>
<br>
Vish, I am seeing the exact same issue as shown below. Instances end up in ERROR state and looking into the nova-network log, I find *no* errors at all, and yet looking at the nova-compute log, I see multiple timeout errors -- all of them trying to RPC while
in the allocate_network method. Always the same method, always the same error, and no errors in nova-network or nova-api (other than just reporting a failed build)<br>
<br>
Any idea on something that may have crept in recently? This wasn't happening a week or so ago, AFAICT.<br>
<br>
Best,<br>
-jay<br>
<br>
On 06/18/2012 06:03 PM, Lillie Ross-CDSR11 wrote:<br>
<blockquote type="cite">I'm receiving RPC timeouts when trying to launch an instance. My<br>
</blockquote>
<blockquote type="cite">installation is the Essex release running on Ubuntu 12.04.<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">When I launch a test image, the launch fails. In my setup, Nova network<br>
</blockquote>
<blockquote type="cite">runs on a controller node, and all compute instances run on separate,<br>
</blockquote>
<blockquote type="cite">dedicated server nodes. The failure is repeatable. Upon examining the<br>
</blockquote>
<blockquote type="cite">various logs, I see the following (see below). Any insight would be welcome.<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">Regards,<br>
</blockquote>
<blockquote type="cite">Ross<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">From 'nova show <instance name>' I read the following:<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">root@cirrus1:~# nova show test<br>
</blockquote>
<blockquote type="cite">+-------------------------------------+-----------------------------------------------------------------------------+<br>
</blockquote>
<blockquote type="cite">| Property | Value |<br>
</blockquote>
<blockquote type="cite">+-------------------------------------+-----------------------------------------------------------------------------+<br>
</blockquote>
<blockquote type="cite">| OS-DCF:diskConfig | MANUAL |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-SRV-ATTR:host | nova8 |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-SRV-ATTR:hypervisor_hostname | None |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-STS:power_state | 0 |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-STS:task_state | networking |<br>
</blockquote>
<blockquote type="cite">| OS-EXT-STS:vm_state | error |<br>
</blockquote>
<blockquote type="cite">| accessIPv4 | |<br>
</blockquote>
<blockquote type="cite">| accessIPv6 | |<br>
</blockquote>
<blockquote type="cite">| config_drive | |<br>
</blockquote>
<blockquote type="cite">| created | 2012-06-18T20:42:56Z |<br>
</blockquote>
<blockquote type="cite">| fault | {u'message': u'Timeout', u'code': 500, u'created':<br>
</blockquote>
<blockquote type="cite">u'2012-06-18T20:43:58Z'} |<br>
</blockquote>
<blockquote type="cite">| flavor | m1.tiny |<br>
</blockquote>
<blockquote type="cite">| hostId | 50272989300483e2b5e5236cd572fef3f9149ae60faa5f5660f8da54 |<br>
</blockquote>
<blockquote type="cite">| id | d569b16f-10a8-4cb8-90a3-d5b664c2322d |<br>
</blockquote>
<blockquote type="cite">| image | tty-linux |<br>
</blockquote>
<blockquote type="cite">| key_name | admin |<br>
</blockquote>
<blockquote type="cite">| metadata | {} |<br>
</blockquote>
<blockquote type="cite">| name | test |<br>
</blockquote>
<blockquote type="cite">| private_0 network | |<br>
</blockquote>
<blockquote type="cite">| status | ERROR |<br>
</blockquote>
<blockquote type="cite">| tenant_id | 1 |<br>
</blockquote>
<blockquote type="cite">| updated | 2012-06-18T20:43:57Z |<br>
</blockquote>
<blockquote type="cite">| user_id | 1 |<br>
</blockquote>
<blockquote type="cite">+-------------------------------------+-----------------------------------------------------------------------------+<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">From the nova-network.log I see the following:<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:36 DEBUG nova.manager [-] Running periodic task<br>
</blockquote>
<blockquote type="cite">VlanManager._disassociate_stale_fixed_ips from (pid=1381) periodic_tasks<br>
</blockquote>
<blockquote type="cite">/usr/lib/python2.7/dist-packages<br>
</blockquote>
<blockquote type="cite">/nova/manager.py:152<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 ERROR nova.rpc.common [-] Timed out waiting for RPC<br>
</blockquote>
<blockquote type="cite">response: timed out<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common Traceback (most recent call last):<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in<br>
</blockquote>
<blockquote type="cite">ensure<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return method(*args, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in<br>
</blockquote>
<blockquote type="cite">_consume<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.connection.drain_events(timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in<br>
</blockquote>
<blockquote type="cite">drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.transport.drain_events(self.connection, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">238, in drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">connection.drain_events(**kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">57, in drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.wait_multi(self.channels.values(), timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">63, in wait_multi<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common chanmap.keys(),<br>
</blockquote>
<blockquote type="cite">allowed_methods, timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">120, in _wait_multiple<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common channel, method_sig, args,<br>
</blockquote>
<blockquote type="cite">content = read_timeout(timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">94, in read_timeout<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.method_reader.read_method()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",<br>
</blockquote>
<blockquote type="cite">line 221, in read_method<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common raise m<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common timeout: timed out<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 DEBUG nova.utils<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] Attempting to grab<br>
</blockquote>
<blockquote type="cite">semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1381) i<br>
</blockquote>
<blockquote type="cite">nner /usr/lib/python2.7/dist-packages/nova/utils.py:927<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 DEBUG nova.utils<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] Got semaphore "get_dhcp"<br>
</blockquote>
<blockquote type="cite">for method "_get_dhcp_ip"... from (pid=1381) inner /usr/lib/p<br>
</blockquote>
<blockquote type="cite">ython2.7/dist-packages/nova/utils.py:931<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 DEBUG nova.utils<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] Attempting to grab<br>
</blockquote>
<blockquote type="cite">semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1381) i<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">and from nova-compute.log I read:<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 ERROR nova.rpc.common<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] Timed out waiting for RPC<br>
</blockquote>
<blockquote type="cite">response: timed out<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common Traceback (most recent call last):<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in<br>
</blockquote>
<blockquote type="cite">ensure<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return method(*args, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in<br>
</blockquote>
<blockquote type="cite">_consume<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.connection.drain_events(timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in<br>
</blockquote>
<blockquote type="cite">drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.transport.drain_events(self.connection, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">238, in drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">connection.drain_events(**kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">57, in drain_events<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.wait_multi(self.channels.values(), timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">63, in wait_multi<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common chanmap.keys(),<br>
</blockquote>
<blockquote type="cite">allowed_methods, timeout=timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">120, in _wait_multiple<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common channel, method_sig, args,<br>
</blockquote>
<blockquote type="cite">content = read_timeout(timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line<br>
</blockquote>
<blockquote type="cite">94, in read_timeout<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common return<br>
</blockquote>
<blockquote type="cite">self.method_reader.read_method()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",<br>
</blockquote>
<blockquote type="cite">line 221, in read_method<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common raise m<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common timeout: timed out<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.rpc.common<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 ERROR nova.compute.manager<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] Instance failed network setup<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] Traceback (most recent call last):<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in<br>
</blockquote>
<blockquote type="cite">_allocate_network<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] requested_networks=requested_networks)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in<br>
</blockquote>
<blockquote type="cite">allocate_for_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] 'args': args})<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] return _get_impl().call(context,<br>
</blockquote>
<blockquote type="cite">topic, msg, timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] return rpc_amqp.call(context,<br>
</blockquote>
<blockquote type="cite">topic, msg, timeout, Connection.pool)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] rv = list(rv)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] self._iterator.next()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in<br>
</blockquote>
<blockquote type="cite">iterconsume<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] yield self.ensure(_error_callback,<br>
</blockquote>
<blockquote type="cite">_consume)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in<br>
</blockquote>
<blockquote type="cite">ensure<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] error_callback(e)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in<br>
</blockquote>
<blockquote type="cite">_error_callback<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] raise rpc_common.Timeout()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d] Timeout: Timeout while waiting on<br>
</blockquote>
<blockquote type="cite">RPC response.<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:57 TRACE nova.compute.manager [instance:<br>
</blockquote>
<blockquote type="cite">d569b16f-10a8-4cb8-90a3-d5b664c2322d]<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 ERROR nova.rpc.amqp<br>
</blockquote>
<blockquote type="cite">[req-16158a6b-f3d6-49f3-977e-3ccfecc791ca 1 1] Exception during message<br>
</blockquote>
<blockquote type="cite">handling<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp Traceback (most recent call last):<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in<br>
</blockquote>
<blockquote type="cite">_process_data<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp rval = node_func(context=ctxt,<br>
</blockquote>
<blockquote type="cite">**node_args)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp return f(*args, **kw)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in<br>
</blockquote>
<blockquote type="cite">decorated_function<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp sys.exc_info())<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp self.gen.next()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in<br>
</blockquote>
<blockquote type="cite">decorated_function<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp return function(self, context,<br>
</blockquote>
<blockquote type="cite">instance_uuid, *args, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 651, in<br>
</blockquote>
<blockquote type="cite">run_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp do_run_instance()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp retval = f(*args, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 650, in<br>
</blockquote>
<blockquote type="cite">do_run_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp self._run_instance(context,<br>
</blockquote>
<blockquote type="cite">instance_uuid, **kwargs)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 451, in<br>
</blockquote>
<blockquote type="cite">_run_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp<br>
</blockquote>
<blockquote type="cite">self._set_instance_error_state(context, instance_uuid)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp self.gen.next()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 424, in<br>
</blockquote>
<blockquote type="cite">_run_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp requested_networks)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in<br>
</blockquote>
<blockquote type="cite">_allocate_network<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp<br>
</blockquote>
<blockquote type="cite">requested_networks=requested_networks)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in<br>
</blockquote>
<blockquote type="cite">allocate_for_instance<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp 'args': args})<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp return _get_impl().call(context,<br>
</blockquote>
<blockquote type="cite">topic, msg, timeout)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp return rpc_amqp.call(context,<br>
</blockquote>
<blockquote type="cite">topic, msg, timeout, Connection.pool)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp rv = list(rv)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp self._iterator.next()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in<br>
</blockquote>
<blockquote type="cite">iterconsume<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp yield<br>
</blockquote>
<blockquote type="cite">self.ensure(_error_callback, _consume)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in<br>
</blockquote>
<blockquote type="cite">ensure<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp error_callback(e)<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp File<br>
</blockquote>
<blockquote type="cite">"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in<br>
</blockquote>
<blockquote type="cite">_error_callback<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp raise rpc_common.Timeout()<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp Timeout: Timeout while waiting<br>
</blockquote>
<blockquote type="cite">on RPC response.<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:43:58 TRACE nova.rpc.amqp<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:44:05 DEBUG nova.manager [-] Running periodic task<br>
</blockquote>
<blockquote type="cite">ComputeManager._publish_service_capabilities from (pid=13262)<br>
</blockquote>
<blockquote type="cite">periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:44:05 DEBUG nova.manager [-] Notifying Schedulers of<br>
</blockquote>
<blockquote type="cite">capabilities ... from (pid=13262) _publish_service_capabilities<br>
</blockquote>
<blockquote type="cite">/usr/lib/python2.7/dist-packages/nova/manager.py:203<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:44:05 DEBUG nova.rpc.amqp [-] Making asynchronous fanout<br>
</blockquote>
<blockquote type="cite">cast... from (pid=13262) fanout_cast<br>
</blockquote>
<blockquote type="cite">/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:354<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:44:05 DEBUG nova.manager [-] Running periodic task<br>
</blockquote>
<blockquote type="cite">ComputeManager._poll_rescued_instances from (pid=13262) periodic_tasks<br>
</blockquote>
<blockquote type="cite">/usr/lib/python2.7/dist-packages/nova/manager.py:152<br>
</blockquote>
<blockquote type="cite">2012-06-18 15:44:05 DEBUG nova.manager [-] Skipping<br>
</blockquote>
<blockquote type="cite">ComputeManager._sync_power_states, 1 ticks left until next run from<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">_______________________________________________<br>
</blockquote>
<blockquote type="cite">Mailing list: <a href="https://launchpad.net/~openstack">
https://launchpad.net/~openstack</a><br>
</blockquote>
<blockquote type="cite">Post to : <a href="mailto:openstack@lists.launchpad.net">
openstack@lists.launchpad.net</a><br>
</blockquote>
<blockquote type="cite">Unsubscribe : <a href="https://launchpad.net/~openstack">
https://launchpad.net/~openstack</a><br>
</blockquote>
<blockquote type="cite">More help : <a href="https://help.launchpad.net/ListHelp">
https://help.launchpad.net/ListHelp</a><br>
</blockquote>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>
</blockquote>
</div>
<br>
</div>
</body>
</html>