<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>From this line:<o:p></o:p></span></p><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common timeout: timed out<o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>It is worth checking to see if you have all your services running, and that they all have the correct Rabbit server configured. Try “nova-manage service list” and make sure you can see all the services you expect to see.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>I don’t see any logs form nova-network, it might be that nova-network is failing in a way that means it never replies the RPC call. Can you get hold of the logs from that component?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>I hope that helps,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>John<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> openstack-bounces+john.garbutt=citrix.com@lists.launchpad.net [mailto:openstack-bounces+john.garbutt=citrix.com@lists.launchpad.net] <b>On Behalf Of </b>Afef MDHAFFAR<br><b>Sent:</b> 10 December 2012 16:59<br><b>To:</b> openstack@lists.launchpad.net<br><b>Subject:</b> [Openstack] can not launch an instance + RPC timeout + nova-network does not start dnsmasq<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi all, <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I installed openstack (folsom release) on ubuntu 12.04 + XCP.<o:p></o:p></p></div><div><p class=MsoNormal>All services seem to be running.<o:p></o:p></p></div><div><p class=MsoNormal>However nova-network does not start dnsmasq -- and the interface xapi0 is missing.<o:p></o:p></p></div><div><p class=MsoNormal>I can not launch an instance , and I got the following logs on nova-compute.<o:p></o:p></p></div><div><p class=MsoNormal>Please try to help to fix this issue.<o:p></o:p></p></div><div><p class=MsoNormal>Thank you,<o:p></o:p></p></div><div><p class=MsoNormal>Afef<o:p></o:p></p></div><div><p class=MsoNormal>------<o:p></o:p></p></div><div><div><p class=MsoNormal>2012-12-10 16:51:14 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:15 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 ERROR nova.openstack.common.rpc.common [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Timed out waiting for RPC response: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common raise m<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common timeout: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 TRACE nova.openstack.common.rpc.common <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 DEBUG nova.openstack.common.rpc.amqp [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Making asynchronous call on conductor ... from (pid=28401) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:359<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:51:37 DEBUG nova.openstack.common.rpc.amqp [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] MSG_ID is 806e29cdea2640b9a03a6990d793c57b from (pid=28401) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:362<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager.publish_service_capabilities from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=28401) publish_service_capabilities /opt/stack/nova/nova/manager.py:257<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=28401) fanout_cast /opt/stack/nova/nova/openstack/common/rpc/amqp.py:392<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_volume_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 7 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager._instance_usage_audit from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.openstack.common.lockutils [-] Got semaphore "compute_resources" for method "update_available_resource"... from (pid=28401) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:15 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.compute.resource_tracker [-] Hypervisor: free ram (MB): 4938 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:322<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.compute.resource_tracker [-] Hypervisor: free disk (GB): 218 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:323<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.compute.resource_tracker [-] Hypervisor: VCPU information unavailable from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:330<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 AUDIT nova.compute.resource_tracker [-] Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 INFO nova.compute.resource_tracker [-] Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 27 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Running periodic task ComputeManager._check_instance_build_time from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 37 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.compute.manager [-] CONF.reclaim_instance_interval <= 0, skipping... from (pid=28401) _reclaim_queued_deletes /opt/stack/nova/nova/compute/manager.py:3215<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:16 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 ERROR nova.openstack.common.rpc.common [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Timed out waiting for RPC response: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common raise m<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common timeout: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 TRACE nova.openstack.common.rpc.common <o:p></o:p></p></div><div><p class=MsoNormal>ERROR:root:Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 605, in _run_instance\n self._start_building(context, instance)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 835, in _start_building\n None))\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 349, in _instance_update\n **kwargs)\n', ' File "/opt/stack/nova/nova/conductor/api.py", line 78, in instance_update\n updates)\n', ' File "/opt/stack/nova/nova/conductor/rpcapi.py", line 48, in instance_update\n updates=updates_p))\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call\n return rpc.call(context, self._get_topic(topic), msg, timeout)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call\n return _get_impl().call(cfg.CONF, context, topic, msg, timeout)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 376, in call\n rv = list(rv)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 337, in __iter__\n self.done()\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 334, in __iter__\n self._iterator.next()\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume\n yield self.ensure(_error_callback, _consume)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure\n error_callback(e)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback\n raise rpc_common.Timeout()\n', 'Timeout: Timeout while waiting on RPC response.\n']<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 DEBUG nova.openstack.common.rpc.amqp [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Making asynchronous call on conductor ... from (pid=28401) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:359<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.rpc.amqp:Making asynchronous call on conductor ...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:52:37 DEBUG nova.openstack.common.rpc.amqp [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] MSG_ID is bca57b2f78434b58beac25467d23177c from (pid=28401) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:362<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.rpc.amqp:MSG_ID is bca57b2f78434b58beac25467d23177c<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager.publish_service_capabilities from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.publish_service_capabilities<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=28401) publish_service_capabilities /opt/stack/nova/nova/manager.py:257<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Notifying Schedulers of capabilities ...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=28401) fanout_cast /opt/stack/nova/nova/openstack/common/rpc/amqp.py:392<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.rpc.amqp:Making asynchronous fanout cast...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._heal_instance_info_cache<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rescued_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_volume_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_volume_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_bandwidth_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 6 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._sync_power_states, 6 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager._instance_usage_audit from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._instance_usage_audit<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.update_available_resource<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:16 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 DEBUG nova.openstack.common.lockutils [-] Got semaphore "compute_resources" for method "update_available_resource"... from (pid=28401) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.lockutils:Got semaphore "compute_resources" for method "update_available_resource"...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 DEBUG nova.compute.resource_tracker [-] Hypervisor: free ram (MB): 4938 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:322<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free ram (MB): 4938<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 DEBUG nova.compute.resource_tracker [-] Hypervisor: free disk (GB): 218 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:323<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free disk (GB): 218<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 DEBUG nova.compute.resource_tracker [-] Hypervisor: VCPU information unavailable from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:330<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:17 AUDIT nova.compute.resource_tracker [-] Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 INFO nova.compute.resource_tracker [-] Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>INFO:nova.compute.resource_tracker:Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rebooting_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 26 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._cleanup_running_deleted_instances, 26 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Running periodic task ComputeManager._check_instance_build_time from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._check_instance_build_time<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 36 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._run_image_cache_manager_pass, 36 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._reclaim_queued_deletes<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.compute.manager [-] CONF.reclaim_instance_interval <= 0, skipping... from (pid=28401) _reclaim_queued_deletes /opt/stack/nova/nova/compute/manager.py:3215<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.manager:CONF.reclaim_instance_interval <= 0, skipping...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._report_driver_status<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 INFO nova.compute.manager [-] Updating host status<o:p></o:p></p></div><div><p class=MsoNormal>INFO:nova.compute.manager:Updating host status<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:18 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_unconfirmed_resizes<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 ERROR nova.openstack.common.rpc.common [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Timed out waiting for RPC response: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common raise m<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common timeout: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common <o:p></o:p></p></div><div><p class=MsoNormal>ERROR:nova.openstack.common.rpc.common:Timed out waiting for RPC response: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common raise m<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common timeout: timed out<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.common <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 ERROR nova.openstack.common.rpc.amqp [req-2c7c1df0-e6f7-4a13-83d6-c51e23d0fcbb admin admin] Exception during message handling<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 115, in wrapped<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp temp_level, payload)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp pass<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 198, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 239, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 227, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 976, in run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp do_run_instance()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 975, in do_run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 649, in _run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 361, in _set_instance_error_state<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp vm_state=vm_states.ERROR)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 349, in _instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/conductor/api.py", line 78, in instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp updates)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/conductor/rpcapi.py", line 48, in instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp updates=updates_p))<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return rpc.call(context, self._get_topic(topic), msg, timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 376, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rv = list(rv)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 337, in __iter__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.done()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 334, in __iter__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self._iterator.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp yield self.ensure(_error_callback, _consume)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp error_callback(e)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp raise rpc_common.Timeout()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response.<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp <o:p></o:p></p></div><div><p class=MsoNormal>ERROR:nova.openstack.common.rpc.amqp:Exception during message handling<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 115, in wrapped<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp temp_level, payload)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp pass<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 198, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 239, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 227, in decorated_function<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 976, in run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp do_run_instance()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 975, in do_run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 649, in _run_instance<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 361, in _set_instance_error_state<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp vm_state=vm_states.ERROR)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 349, in _instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp **kwargs)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/conductor/api.py", line 78, in instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp updates)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/conductor/rpcapi.py", line 48, in instance_update<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp updates=updates_p))<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return rpc.call(context, self._get_topic(topic), msg, timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 376, in call<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp rv = list(rv)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 337, in __iter__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.done()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self.gen.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 334, in __iter__<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp self._iterator.next()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp yield self.ensure(_error_callback, _consume)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp error_callback(e)<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp raise rpc_common.Timeout()<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response.<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:53:37 TRACE nova.openstack.common.rpc.amqp <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager.publish_service_capabilities from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.publish_service_capabilities<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=28401) publish_service_capabilities /opt/stack/nova/nova/manager.py:257<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Notifying Schedulers of capabilities ...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=28401) fanout_cast /opt/stack/nova/nova/openstack/common/rpc/amqp.py:392<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.rpc.amqp:Making asynchronous fanout cast...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._heal_instance_info_cache<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rescued_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_volume_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_volume_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_bandwidth_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 5 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._sync_power_states, 5 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager._instance_usage_audit from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._instance_usage_audit<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.update_available_resource<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:18 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.openstack.common.lockutils [-] Got semaphore "compute_resources" for method "update_available_resource"... from (pid=28401) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.lockutils:Got semaphore "compute_resources" for method "update_available_resource"...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.compute.resource_tracker [-] Hypervisor: free ram (MB): 4938 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:322<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free ram (MB): 4938<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.compute.resource_tracker [-] Hypervisor: free disk (GB): 218 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:323<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free disk (GB): 218<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.compute.resource_tracker [-] Hypervisor: VCPU information unavailable from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:330<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 AUDIT nova.compute.resource_tracker [-] Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 INFO nova.compute.resource_tracker [-] Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>INFO:nova.compute.resource_tracker:Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rebooting_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 25 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._cleanup_running_deleted_instances, 25 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Running periodic task ComputeManager._check_instance_build_time from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._check_instance_build_time<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 35 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._run_image_cache_manager_pass, 35 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._reclaim_queued_deletes<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.compute.manager [-] CONF.reclaim_instance_interval <= 0, skipping... from (pid=28401) _reclaim_queued_deletes /opt/stack/nova/nova/compute/manager.py:3215<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.manager:CONF.reclaim_instance_interval <= 0, skipping...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._report_driver_status<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:54:19 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_unconfirmed_resizes<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager.publish_service_capabilities from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.publish_service_capabilities<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=28401) publish_service_capabilities /opt/stack/nova/nova/manager.py:257<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Notifying Schedulers of capabilities ...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous fanout cast... from (pid=28401) fanout_cast /opt/stack/nova/nova/openstack/common/rpc/amqp.py:392<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.rpc.amqp:Making asynchronous fanout cast...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._heal_instance_info_cache<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rescued_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_volume_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_volume_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_bandwidth_usage<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 4 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._sync_power_states, 4 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager._instance_usage_audit from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._instance_usage_audit<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager.update_available_resource<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:19 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 DEBUG nova.openstack.common.lockutils [-] Got semaphore "compute_resources" for method "update_available_resource"... from (pid=28401) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.openstack.common.lockutils:Got semaphore "compute_resources" for method "update_available_resource"...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Auditing locally available compute resources<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 DEBUG nova.compute.resource_tracker [-] Hypervisor: free ram (MB): 4938 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:322<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free ram (MB): 4938<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 DEBUG nova.compute.resource_tracker [-] Hypervisor: free disk (GB): 218 from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:323<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: free disk (GB): 218<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:20 DEBUG nova.compute.resource_tracker [-] Hypervisor: VCPU information unavailable from (pid=28401) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:330<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.resource_tracker:Hypervisor: VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free ram (MB): 7679<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free disk (GB): 225<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 AUDIT nova.compute.resource_tracker [-] Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>AUDIT:nova.compute.resource_tracker:Free VCPU information unavailable<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 INFO nova.compute.resource_tracker [-] Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>INFO:nova.compute.resource_tracker:Compute_service record updated for openstackDomU <o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_rebooting_instances<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 24 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._cleanup_running_deleted_instances, 24 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Running periodic task ComputeManager._check_instance_build_time from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._check_instance_build_time<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 34 ticks left until next run from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:167<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Skipping ComputeManager._run_image_cache_manager_pass, 34 ticks left until next run<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._reclaim_queued_deletes<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.compute.manager [-] CONF.reclaim_instance_interval <= 0, skipping... from (pid=28401) _reclaim_queued_deletes /opt/stack/nova/nova/compute/manager.py:3215<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.compute.manager:CONF.reclaim_instance_interval <= 0, skipping...<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._report_driver_status<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 INFO nova.compute.manager [-] Updating host status<o:p></o:p></p></div><div><p class=MsoNormal>INFO:nova.compute.manager:Updating host status<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=28401) update_status /opt/stack/nova/nova/virt/xenapi/host.py:150<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.virt.xenapi.host:Updating host stats<o:p></o:p></p></div><div><p class=MsoNormal>2012-12-10 16:55:21 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=28401) periodic_tasks /opt/stack/nova/nova/manager.py:172<o:p></o:p></p></div><div><p class=MsoNormal>DEBUG:nova.manager:Running periodic task ComputeManager._poll_unconfirmed_resizes<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></body></html>