thanks again for your answer. I restarted nova-network nut I still get the same problem :(<div><br><div class="gmail_quote">2012/11/22 Mohammed Naser <span dir="ltr"><<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<div><br></div><div>Can you try restarting nova-network? It seems that it's not properly responding to the command which is why this happens.</div>
<div><br></div><div>Thanks,</div><div>Mohammed</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Nov 21, 2012 at 6:02 PM, Afef MDHAFFAR <span dir="ltr"><<a href="mailto:afef.mdhaffar@gmail.com" target="_blank">afef.mdhaffar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
thank you for the fast answer. nova-network is running.<div>I am really going to get crazy!. I do not understand, why it does not work anymore???<br><br><div class="gmail_quote">2012/11/21 Mohammed Naser <span dir="ltr"><<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Adef,<div><br></div><div>Please check that your nova-network instance is running, look like it's not (or not responding in a timely fashion)</div>
<div><br></div><div>Regards,</div><div>Mohammed</div><div class="gmail_extra">
<br><br><div class="gmail_quote"><div><div>On Wed, Nov 21, 2012 at 5:52 PM, Afef MDHAFFAR <span dir="ltr"><<a href="mailto:afef.mdhaffar@gmail.com" target="_blank">afef.mdhaffar@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
Hi all,<div><br></div><div>I installed openstack with xcp on 7 nodes, while using devsstack. It was working and I could launch an instance.</div><div>Suddenly, launching instance fails and I got this error message. I need an urgent help, please.</div>
<div>2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=9785) periodic_tasks /opt/stack/nova/nova/manager.py:171</div><div>2012-11-21 22:45:56 INFO nova.compute.manager [-] Updating host status</div>
<div>2012-11-21 22:45:56 DEBUG nova.virt.xenapi.host [-] Updating host stats from (pid=9785) update_status /opt/stack/nova/nova/virt/xenapi/host.py:156</div><div>2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=9785) periodic_tasks /opt/stack/nova/nova/manager.py:171</div>
<div>2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common raise m</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout: timed out</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common </div><div>2012-11-21 22:46:16 ERROR nova.compute.manager [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Instance failed network setup</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] requested_networks=requested_networks)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/network/api.py", line 44, in wrapper</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] res = f(self, context, *args, **kwargs)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] nw_info = self.network_rpcapi.allocate_for_instance(context, **args)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] vpn=vpn, requested_networks=requested_networks))</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] return rpc.call(context, self._get_topic(topic), msg, timeout)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] rpc_amqp.get_connection_pool(conf, Connection))</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] rv = list(rv)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] self.done()</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] self.gen.next()</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] self._iterator.next()</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] yield self.ensure(_error_callback, _consume)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] error_callback(e)</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] raise rpc_common.Timeout()</div><div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Timeout: Timeout while waiting on RPC response.</div>
<div>2012-11-21 22:46:16 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] </div><div>2012-11-21 22:46:16 DEBUG nova.openstack.common.lockutils [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore "compute_resources" for method "abort"... from (pid=9785) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185</div>
<div>2012-11-21 22:46:16 DEBUG nova.compute.claims [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Aborting claim: [Claim: 512 MB memory, 0 GB disk, 1 VCPUS] from (pid=9785) abort /opt/stack/nova/nova/compute/claims.py:91</div>
<div>2012-11-21 22:46:16 DEBUG nova.compute.manager [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Deallocating network for instance from (pid=9785) _deallocate_network /opt/stack/nova/nova/compute/manager.py:866</div>
<div>2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous call on network ... from (pid=9785) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:352</div><div>2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is eb98cef4e39143459409e551393e9980 from (pid=9785) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:355</div>
<div>2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'KeystoneAdmin', u'admin', u'KeystoneServiceAdmin'], u'_msg_id': u'eb98cef4e39143459409e551393e9980', u'_context_quota_class': None, u'_context_request_id': u'req-b091ff10-7159-45bd-a5cf-cd33af605303', u'_context_service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f</a>', u'internalURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f</a>', u'id': u'd4e294422e03492eafc09ad6f5766935'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:3333" target="_blank">http://10.0.1.15:3333</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:3333" target="_blank">http://10.0.1.15:3333</a>', u'internalURL': u'<a href="http://10.0.1.15:3333" target="_blank">http://10.0.1.15:3333</a>', u'id': u'9561b04884e743c1a20ae8ed41c11618'}], u'type': u's3', u'name': u's3'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:9292" target="_blank">http://10.0.1.15:9292</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:9292" target="_blank">http://10.0.1.15:9292</a>', u'internalURL': u'<a href="http://10.0.1.15:9292" target="_blank">http://10.0.1.15:9292</a>', u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f</a>', u'internalURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f</a>', u'id': u'cc784ab5f2664a398cce7352f71b31a7'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8773/services/Admin" target="_blank">http://10.0.1.15:8773/services/Admin</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:8773/services/Cloud" target="_blank">http://10.0.1.15:8773/services/Cloud</a>', u'internalURL': u'<a href="http://10.0.1.15:8773/services/Cloud" target="_blank">http://10.0.1.15:8773/services/Cloud</a>', u'id': u'4010dc2bbadf42a79f18af6f19e6efe7'}], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:35357/v2.0" target="_blank">http://10.0.1.15:35357/v2.0</a>', u'region': u'RegionOne', u'publicURL': u'<a href="http://10.0.1.15:5000/v2.0" target="_blank">http://10.0.1.15:5000/v2.0</a>', u'internalURL': u'<a href="http://10.0.1.15:5000/v2.0" target="_blank">http://10.0.1.15:5000/v2.0</a>', u'id': u'759f932fafea448299c9461c8a56cdf1'}], u'type': u'identity', u'name': u'keystone'}], u'_context_user_name': u'admin', u'_context_auth_token': '<SANITIZED>', u'args': {u'instance_id': 10, u'host': None, u'project_id': u'bca2f35a23ab466ca6b160648461015f'}, u'_context_instance_lock_checked': False, u'_context_project_name': u'admin', u'_context_is_admin': True, u'version': u'1.0', u'_context_project_id': u'bca2f35a23ab466ca6b160648461015f', u'_context_timestamp': u'2012-11-21T22:45:11.517140', u'_context_read_deleted': u'no', u'_context_user_id': u'1745eb1977bc4bcaac3d44e3fbee7647', u'method': u'deallocate_for_instance', u'_context_remote_address': u'10.0.1.15'} from (pid=9786) _safe_log /opt/stack/nova/nova/openstack/common/rpc/common.py:195</div>
<div>2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': u'admin', 'user_id': u'1745eb1977bc4bcaac3d44e3fbee7647', 'roles': [u'KeystoneAdmin', u'admin', u'KeystoneServiceAdmin'], 'timestamp': u'2012-11-21T22:45:11.517140', 'auth_token': '<SANITIZED>', 'remote_address': u'10.0.1.15', 'quota_class': None, 'is_admin': True, 'service_catalog': [{u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f</a>', u'region': u'RegionOne', u'id': u'd4e294422e03492eafc09ad6f5766935', u'internalURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f</a>', u'publicURL': u'<a href="http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f'%7D" target="_blank">http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f'}</a>], u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}, {u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:3333" target="_blank">http://10.0.1.15:3333</a>', u'region': u'RegionOne', u'id': u'9561b04884e743c1a20ae8ed41c11618', u'internalURL': u'<a href="http://10.0.1.15:3333" target="_blank">http://10.0.1.15:3333</a>', u'publicURL': u'http://10.0.1.15:3333'}], u'endpoints_links': [], u'type': u's3', u'name': u's3'}, {u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:9292" target="_blank">http://10.0.1.15:9292</a>', u'region': u'RegionOne', u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77', u'internalURL': u'<a href="http://10.0.1.15:9292" target="_blank">http://10.0.1.15:9292</a>', u'publicURL': u'http://10.0.1.15:9292'}], u'endpoints_links': [], u'type': u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f</a>', u'region': u'RegionOne', u'id': u'cc784ab5f2664a398cce7352f71b31a7', u'internalURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f</a>', u'publicURL': u'<a href="http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f'%7D" target="_blank">http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f'}</a>], u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'}, {u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:8773/services/Admin" target="_blank">http://10.0.1.15:8773/services/Admin</a>', u'region': u'RegionOne', u'id': u'4010dc2bbadf42a79f18af6f19e6efe7', u'internalURL': u'<a href="http://10.0.1.15:8773/services/Cloud" target="_blank">http://10.0.1.15:8773/services/Cloud</a>', u'publicURL': u'<a href="http://10.0.1.15:8773/services/Cloud'%7D" target="_blank">http://10.0.1.15:8773/services/Cloud'}</a>], u'endpoints_links': [], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL': u'<a href="http://10.0.1.15:35357/v2.0" target="_blank">http://10.0.1.15:35357/v2.0</a>', u'region': u'RegionOne', u'id': u'759f932fafea448299c9461c8a56cdf1', u'internalURL': u'<a href="http://10.0.1.15:5000/v2.0" target="_blank">http://10.0.1.15:5000/v2.0</a>', u'publicURL': u'<a href="http://10.0.1.15:5000/v2.0'%7D" target="_blank">http://10.0.1.15:5000/v2.0'}</a>], u'endpoints_links': [], u'type': u'identity', u'name': u'keystone'}], 'request_id': u'req-b091ff10-7159-45bd-a5cf-cd33af605303', 'instance_lock_checked': False, 'project_id': u'bca2f35a23ab466ca6b160648461015f', 'user_name': u'admin', 'read_deleted': u'no'} from (pid=9786) _safe_log /opt/stack/nova/nova/openstack/common/rpc/common.py:195</div>
<div>2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in ensure</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in _consume</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = self.read_timeout(timeout)</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common raise m</div>
<div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout: timed out</div><div>2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common </div><div>Traceback (most recent call last):</div><div> File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl</div>
<div> func(*args, **kwargs)</div><div> File "/opt/stack/nova/nova/network/rpcapi.py", line 236, in _rpc_allocate_fixed_ip</div><div> topic=rpc.queue_get_for(ctxt, self.topic, host))</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call</div>
<div> return rpc.call(context, self._get_topic(topic), msg, timeout)</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call</div><div> return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div>
<div> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call</div><div> rpc_amqp.get_connection_pool(conf, Connection))</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div> rv = list(rv)</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div><div> self.done()</div><div> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div> self.gen.next()</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div><div> self._iterator.next()</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume</div>
<div> yield self.ensure(_error_callback, _consume)</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure</div><div> error_callback(e)</div><div> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback</div>
<div> raise rpc_common.Timeout()</div><div>Timeout: Timeout while waiting on RPC response.</div><div>2012-11-21 22:46:16 DEBUG nova.network.manager [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] network deallocation for instance from (pid=9786) deallocate_for_instance /opt/stack/nova/nova/network/manager.py:1093</div>
<div>2012-11-21 22:46:17 ERROR nova.compute.manager [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Error trying to reschedule</div><div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):</div>
<div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/compute/manager.py", line 641, in _reschedule_or_reraise</div><div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] task_state)</div>
<div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] File "/opt/stack/nova/nova/compute/manager.py", line 659, in _reschedule</div><div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] retry = filter_properties.get('retry', None)</div>
<div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] AttributeError: 'unicode' object has no attribute 'get'</div><div>2012-11-21 22:46:17 TRACE nova.compute.manager [instance: 552cd71c-5e94-4c3a-b431-9244090d1248] </div>
<div>2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore "compute_resources" for method "update_usage"... from (pid=9785) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185</div>
<div>2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore "compute_resources" for method "update_usage"... from (pid=9785) inner /opt/stack/nova/nova/openstack/common/lockutils.py:185</div>
<div>2012-11-21 22:46:17 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in _process_data</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 115, in wrapped</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp temp_level, payload)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 176, in decorated_function</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp pass</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>
2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 162, in decorated_function</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 191, in decorated_function</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 933, in run_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp do_run_instance()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 932, in do_run_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 608, in _run_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 596, in _run_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp is_first_time, request_spec, filter_properties)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 577, in _run_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp requested_networks)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp requested_networks=requested_networks)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/api.py", line 44, in wrapper</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp res = f(self, context, *args, **kwargs)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp nw_info = self.network_rpcapi.allocate_for_instance(context, **args)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp vpn=vpn, requested_networks=requested_networks))</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return rpc.call(context, self._get_topic(topic), msg, timeout)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp rv = list(rv)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.done()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self._iterator.next()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in iterconsume</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp yield self.ensure(_error_callback, _consume)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in ensure</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp error_callback(e)</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in _error_callback</div>
<div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp raise rpc_common.Timeout()</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response.</div><div>2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp </div>
<div>2012-11-21 22:46:42 DEBUG nova.manager [-] Running periodic task SchedulerManager._expire_reservations from (pid=9790) periodic_tasks /opt/stack/nova/nova/manager.py:171</div><div>2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task FlatDHCPManager.publish_service_capabilities from (pid=9786) periodic_tasks /opt/stack/nova/nova/manager.py:171</div>
<div>2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips from (pid=9786) periodic_tasks /opt/stack/nova/nova/manager.py:171</div><div>2012-11-21 22:46:56 DEBUG nova.manager [-] Running periodic task ComputeManager.publish_service_capabilities from (pid=9785) periodic_tasks /opt/stack/nova/nova/manager.py:171</div>
<div>2012-11-21 22:46:56 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=9785) publish_service_capabilities /opt/stack/nova/nova/manager.py:256</div><div> <br><br><br>
</div>
<br></div></div>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><span><font color="#888888"><div><br></div>-- <br><span style="font-family:Arial;font-size:8pt"><span style="color:rgb(70,130,180);font-size:10pt">Mohammed Naser</span> — <span style="font-weight:bold">vexxhost</span><br>
</span><div><span style="font-family:Arial;font-size:8pt"><span><span style="color:rgb(220,220,220)">-----------------------------------------------------</span></span></span></div><div><span style="font-family:Arial;font-size:11px;font-weight:bold">D. </span><span style="font-family:Arial;font-size:11px"><a href="tel:514-316-8872" value="+15143168872" target="_blank">514-316-8872</a></span></div>
<div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">D.</span> <a href="tel:800-910-1726%20ext.%20200" value="+18009101726" target="_blank">800-910-1726 ext. 200</a></span></div><div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">E. </span><a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a></span></div>
<div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">W. </span><a href="http://vexxhost.com/" target="_blank">http://vexxhost.com</a></span></div><br>
</font></span></font></span></div><span><font color="#888888">
</font></span></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br>Afef MDHAFFAR<br><a href="http://www.redcad.org/members/mdhaffar/" target="_blank">http://www.redcad.org/members/mdhaffar/</a><br>
<br><br><br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><span style="font-family:Arial;font-size:8pt"><span style="color:rgb(70,130,180);font-size:10pt">Mohammed Naser</span> — <span style="font-weight:bold">vexxhost</span><br>
</span><div><span style="font-family:Arial;font-size:8pt"><span><span style="color:rgb(220,220,220)">-----------------------------------------------------</span></span></span></div><div><span style="font-family:Arial;font-size:11px;font-weight:bold">D. </span><span style="font-family:Arial;font-size:11px"><a href="tel:514-316-8872" value="+15143168872" target="_blank">514-316-8872</a></span></div>
<div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">D.</span> <a href="tel:800-910-1726%20ext.%20200" value="+18009101726" target="_blank">800-910-1726 ext. 200</a></span></div><div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">E. </span><a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a></span></div>
<div><span style="font-family:Arial;font-size:11px"><span style="font-weight:bold">W. </span><a href="http://vexxhost.com/" target="_blank">http://vexxhost.com</a></span></div><br>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Afef MDHAFFAR<br><a href="http://www.redcad.org/members/mdhaffar/">http://www.redcad.org/members/mdhaffar/</a><br><br><br><br>
</div>