[Openstack] Timed out waiting for RPC response: timed out
Afef MDHAFFAR
afef.mdhaffar at gmail.com
Wed Nov 21 23:08:28 UTC 2012
thanks again for your answer. I restarted nova-network nut I still get the
same problem :(
2012/11/22 Mohammed Naser <mnaser at vexxhost.com>
> Hi,
>
> Can you try restarting nova-network? It seems that it's not properly
> responding to the command which is why this happens.
>
> Thanks,
> Mohammed
>
>
> On Wed, Nov 21, 2012 at 6:02 PM, Afef MDHAFFAR <afef.mdhaffar at gmail.com>wrote:
>
>> thank you for the fast answer. nova-network is running.
>> I am really going to get crazy!. I do not understand, why it does not
>> work anymore???
>>
>> 2012/11/21 Mohammed Naser <mnaser at vexxhost.com>
>>
>>> Hi Adef,
>>>
>>> Please check that your nova-network instance is running, look like it's
>>> not (or not responding in a timely fashion)
>>>
>>> Regards,
>>> Mohammed
>>>
>>>
>>> On Wed, Nov 21, 2012 at 5:52 PM, Afef MDHAFFAR <afef.mdhaffar at gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I installed openstack with xcp on 7 nodes, while using devsstack. It
>>>> was working and I could launch an instance.
>>>> Suddenly, launching instance fails and I got this error message. I need
>>>> an urgent help, please.
>>>> 2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task
>>>> ComputeManager._report_driver_status from (pid=9785) periodic_tasks
>>>> /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:45:56 INFO nova.compute.manager [-] Updating host status
>>>> 2012-11-21 22:45:56 DEBUG nova.virt.xenapi.host [-] Updating host stats
>>>> from (pid=9785) update_status /opt/stack/nova/nova/virt/xenapi/host.py:156
>>>> 2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task
>>>> ComputeManager._poll_unconfirmed_resizes from (pid=9785) periodic_tasks
>>>> /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed
>>>> out waiting for RPC response: timed out
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback
>>>> (most recent call last):
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in
>>>> ensure
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> method(*args, **kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in
>>>> _consume
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.connection.drain_events(timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in
>>>> drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.transport.drain_events(self.connection, **kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200,
>>>> in drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> connection.drain_events(**kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50,
>>>> in drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.wait_multi(self.channels.values(), timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56,
>>>> in wait_multi
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>> chanmap.keys(), allowed_methods, timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104,
>>>> in _wait_multiple
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common channel,
>>>> method_sig, args, content = self.read_timeout(timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86,
>>>> in read_timeout
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.method_reader.read_method()
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>>>> line 215, in read_method
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common raise m
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout:
>>>> timed out
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>> 2012-11-21 22:46:16 ERROR nova.compute.manager
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Instance failed network setup
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>> requested_networks=requested_networks)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/network/api.py", line 44, in wrapper
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] res = f(self, context, *args,
>>>> **kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] nw_info =
>>>> self.network_rpcapi.allocate_for_instance(context, **args)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] vpn=vpn,
>>>> requested_networks=requested_networks))
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] return rpc.call(context,
>>>> self._get_topic(topic), msg, timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] return _get_impl().call(cfg.CONF,
>>>> context, topic, msg, timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>> rpc_amqp.get_connection_pool(conf, Connection))
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] rv = list(rv)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] self.done()
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] self.gen.next()
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] self._iterator.next()
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in
>>>> iterconsume
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] yield
>>>> self.ensure(_error_callback, _consume)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in
>>>> ensure
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] error_callback(e)
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in
>>>> _error_callback
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] raise rpc_common.Timeout()
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Timeout: Timeout while waiting on RPC
>>>> response.
>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.lockutils
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>> "compute_resources" for method "abort"... from (pid=9785) inner
>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>> 2012-11-21 22:46:16 DEBUG nova.compute.claims
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Aborting claim: [Claim: 512 MB
>>>> memory, 0 GB disk, 1 VCPUS] from (pid=9785) abort
>>>> /opt/stack/nova/nova/compute/claims.py:91
>>>> 2012-11-21 22:46:16 DEBUG nova.compute.manager
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Deallocating network for instance
>>>> from (pid=9785) _deallocate_network
>>>> /opt/stack/nova/nova/compute/manager.py:866
>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] Making
>>>> asynchronous call on network ... from (pid=9785) multicall
>>>> /opt/stack/nova/nova/openstack/common/rpc/amqp.py:352
>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is
>>>> eb98cef4e39143459409e551393e9980 from (pid=9785) multicall
>>>> /opt/stack/nova/nova/openstack/common/rpc/amqp.py:355
>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] received
>>>> {u'_context_roles': [u'KeystoneAdmin', u'admin', u'KeystoneServiceAdmin'],
>>>> u'_msg_id': u'eb98cef4e39143459409e551393e9980', u'_context_quota_class':
>>>> None, u'_context_request_id': u'req-b091ff10-7159-45bd-a5cf-cd33af605303',
>>>> u'_context_service_catalog': [{u'endpoints_links': [], u'endpoints':
>>>> [{u'adminURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'region':
>>>> u'RegionOne', u'publicURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>> u'internalURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'id':
>>>> u'd4e294422e03492eafc09ad6f5766935'}], u'type': u'compute', u'name':
>>>> u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:3333', u'region': u'RegionOne', u'publicURL': u'
>>>> http://10.0.1.15:3333', u'internalURL': u'http://10.0.1.15:3333',
>>>> u'id': u'9561b04884e743c1a20ae8ed41c11618'}], u'type': u's3', u'name':
>>>> u's3'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:9292', u'region': u'RegionOne', u'publicURL': u'
>>>> http://10.0.1.15:9292', u'internalURL': u'http://10.0.1.15:9292',
>>>> u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77'}], u'type': u'image', u'name':
>>>> u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'region':
>>>> u'RegionOne', u'publicURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>> u'internalURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'id':
>>>> u'cc784ab5f2664a398cce7352f71b31a7'}], u'type': u'volume', u'name':
>>>> u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:8773/services/Admin', u'region': u'RegionOne',
>>>> u'publicURL': u'http://10.0.1.15:8773/services/Cloud', u'internalURL':
>>>> u'http://10.0.1.15:8773/services/Cloud', u'id':
>>>> u'4010dc2bbadf42a79f18af6f19e6efe7'}], u'type': u'ec2', u'name': u'ec2'},
>>>> {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:35357/v2.0', u'region': u'RegionOne', u'publicURL': u'
>>>> http://10.0.1.15:5000/v2.0', u'internalURL': u'
>>>> http://10.0.1.15:5000/v2.0', u'id':
>>>> u'759f932fafea448299c9461c8a56cdf1'}], u'type': u'identity', u'name':
>>>> u'keystone'}], u'_context_user_name': u'admin', u'_context_auth_token':
>>>> '<SANITIZED>', u'args': {u'instance_id': 10, u'host': None, u'project_id':
>>>> u'bca2f35a23ab466ca6b160648461015f'}, u'_context_instance_lock_checked':
>>>> False, u'_context_project_name': u'admin', u'_context_is_admin': True,
>>>> u'version': u'1.0', u'_context_project_id':
>>>> u'bca2f35a23ab466ca6b160648461015f', u'_context_timestamp':
>>>> u'2012-11-21T22:45:11.517140', u'_context_read_deleted': u'no',
>>>> u'_context_user_id': u'1745eb1977bc4bcaac3d44e3fbee7647', u'method':
>>>> u'deallocate_for_instance', u'_context_remote_address': u'10.0.1.15'} from
>>>> (pid=9786) _safe_log /opt/stack/nova/nova/openstack/common/rpc/common.py:195
>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] unpacked
>>>> context: {'project_name': u'admin', 'user_id':
>>>> u'1745eb1977bc4bcaac3d44e3fbee7647', 'roles': [u'KeystoneAdmin', u'admin',
>>>> u'KeystoneServiceAdmin'], 'timestamp': u'2012-11-21T22:45:11.517140',
>>>> 'auth_token': '<SANITIZED>', 'remote_address': u'10.0.1.15', 'quota_class':
>>>> None, 'is_admin': True, 'service_catalog': [{u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'region':
>>>> u'RegionOne', u'id': u'd4e294422e03492eafc09ad6f5766935', u'internalURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>> u'publicURL': u'
>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f'}],
>>>> u'endpoints_links': [], u'type': u'compute', u'name': u'nova'},
>>>> {u'endpoints': [{u'adminURL': u'http://10.0.1.15:3333', u'region':
>>>> u'RegionOne', u'id': u'9561b04884e743c1a20ae8ed41c11618', u'internalURL': u'
>>>> http://10.0.1.15:3333', u'publicURL': u'http://10.0.1.15:3333'}],
>>>> u'endpoints_links': [], u'type': u's3', u'name': u's3'}, {u'endpoints':
>>>> [{u'adminURL': u'http://10.0.1.15:9292', u'region': u'RegionOne',
>>>> u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77', u'internalURL': u'
>>>> http://10.0.1.15:9292', u'publicURL': u'http://10.0.1.15:9292'}],
>>>> u'endpoints_links': [], u'type': u'image', u'name': u'glance'},
>>>> {u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'region':
>>>> u'RegionOne', u'id': u'cc784ab5f2664a398cce7352f71b31a7', u'internalURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>> u'publicURL': u'
>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f'}],
>>>> u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'},
>>>> {u'endpoints': [{u'adminURL': u'http://10.0.1.15:8773/services/Admin',
>>>> u'region': u'RegionOne', u'id': u'4010dc2bbadf42a79f18af6f19e6efe7',
>>>> u'internalURL': u'http://10.0.1.15:8773/services/Cloud', u'publicURL':
>>>> u'http://10.0.1.15:8773/services/Cloud'}], u'endpoints_links': [],
>>>> u'type': u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL': u'
>>>> http://10.0.1.15:35357/v2.0', u'region': u'RegionOne', u'id':
>>>> u'759f932fafea448299c9461c8a56cdf1', u'internalURL': u'
>>>> http://10.0.1.15:5000/v2.0', u'publicURL': u'
>>>> http://10.0.1.15:5000/v2.0'}], u'endpoints_links': [], u'type':
>>>> u'identity', u'name': u'keystone'}], 'request_id':
>>>> u'req-b091ff10-7159-45bd-a5cf-cd33af605303', 'instance_lock_checked':
>>>> False, 'project_id': u'bca2f35a23ab466ca6b160648461015f', 'user_name':
>>>> u'admin', 'read_deleted': u'no'} from (pid=9786) _safe_log
>>>> /opt/stack/nova/nova/openstack/common/rpc/common.py:195
>>>> 2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed
>>>> out waiting for RPC response: timed out
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback
>>>> (most recent call last):
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in
>>>> ensure
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> method(*args, **kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in
>>>> _consume
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.connection.drain_events(timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in
>>>> drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.transport.drain_events(self.connection, **kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200,
>>>> in drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> connection.drain_events(**kwargs)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50,
>>>> in drain_events
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.wait_multi(self.channels.values(), timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56,
>>>> in wait_multi
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>> chanmap.keys(), allowed_methods, timeout=timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104,
>>>> in _wait_multiple
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common channel,
>>>> method_sig, args, content = self.read_timeout(timeout)
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86,
>>>> in read_timeout
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common return
>>>> self.method_reader.read_method()
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common File
>>>> "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>>>> line 215, in read_method
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common raise m
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout:
>>>> timed out
>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>> Traceback (most recent call last):
>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
>>>> line 80, in _spawn_n_impl
>>>> func(*args, **kwargs)
>>>> File "/opt/stack/nova/nova/network/rpcapi.py", line 236, in
>>>> _rpc_allocate_fixed_ip
>>>> topic=rpc.queue_get_for(ctxt, self.topic, host))
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80,
>>>> in call
>>>> return rpc.call(context, self._get_topic(topic), msg, timeout)
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line
>>>> 108, in call
>>>> return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>> 754, in call
>>>> rpc_amqp.get_connection_pool(conf, Connection))
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369,
>>>> in call
>>>> rv = list(rv)
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330,
>>>> in __iter__
>>>> self.done()
>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> self.gen.next()
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327,
>>>> in __iter__
>>>> self._iterator.next()
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>> 635, in iterconsume
>>>> yield self.ensure(_error_callback, _consume)
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>> 555, in ensure
>>>> error_callback(e)
>>>> File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>> 616, in _error_callback
>>>> raise rpc_common.Timeout()
>>>> Timeout: Timeout while waiting on RPC response.
>>>> 2012-11-21 22:46:16 DEBUG nova.network.manager
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] network deallocation for instance
>>>> from (pid=9786) deallocate_for_instance
>>>> /opt/stack/nova/nova/network/manager.py:1093
>>>> 2012-11-21 22:46:17 ERROR nova.compute.manager
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Error trying to reschedule
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 641, in
>>>> _reschedule_or_reraise
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] task_state)
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 659, in _reschedule
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] retry =
>>>> filter_properties.get('retry', None)
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] AttributeError: 'unicode' object has
>>>> no attribute 'get'
>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>> 2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>> "compute_resources" for method "update_usage"... from (pid=9785) inner
>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>> 2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils
>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>> "compute_resources" for method "update_usage"... from (pid=9785) inner
>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>> 2012-11-21 22:46:17 ERROR nova.openstack.common.rpc.amqp [-] Exception
>>>> during message handling
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Traceback
>>>> (most recent call last):
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in
>>>> _process_data
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp rval =
>>>> self.proxy.dispatch(ctxt, version, method, **args)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in
>>>> dispatch
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> getattr(proxyobj, method)(ctxt, **kwargs)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/exception.py", line 115, in wrapped
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> temp_level, payload)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self.gen.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/exception.py", line 90, in wrapped
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> f(*args, **kw)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 176, in decorated_function
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp pass
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self.gen.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 162, in decorated_function
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> function(self, context, *args, **kwargs)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> kwargs['instance']['uuid'], e, sys.exc_info())
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self.gen.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 191, in decorated_function
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> function(self, context, *args, **kwargs)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 933, in run_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> do_run_instance()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp retval =
>>>> f(*args, **kwargs)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 932, in do_run_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> admin_password, is_first_time, instance)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 608, in _run_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self._set_instance_error_state(context, instance['uuid'])
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self.gen.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 596, in _run_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> is_first_time, request_spec, filter_properties)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 577, in _run_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> requested_networks)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> requested_networks=requested_networks)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/network/api.py", line 44, in wrapper
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp res =
>>>> f(self, context, *args, **kwargs)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp nw_info =
>>>> self.network_rpcapi.allocate_for_instance(context, **args)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp vpn=vpn,
>>>> requested_networks=requested_networks))
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> rpc.call(context, self._get_topic(topic), msg, timeout)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp return
>>>> _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> rpc_amqp.get_connection_pool(conf, Connection))
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp rv =
>>>> list(rv)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp self.done()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self.gen.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> self._iterator.next()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in
>>>> iterconsume
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp yield
>>>> self.ensure(_error_callback, _consume)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in
>>>> ensure
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> error_callback(e)
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp File
>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in
>>>> _error_callback
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp raise
>>>> rpc_common.Timeout()
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Timeout:
>>>> Timeout while waiting on RPC response.
>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>> 2012-11-21 22:46:42 DEBUG nova.manager [-] Running periodic task
>>>> SchedulerManager._expire_reservations from (pid=9790) periodic_tasks
>>>> /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task
>>>> FlatDHCPManager.publish_service_capabilities from (pid=9786) periodic_tasks
>>>> /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task
>>>> FlatDHCPManager._disassociate_stale_fixed_ips from (pid=9786)
>>>> periodic_tasks /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:46:56 DEBUG nova.manager [-] Running periodic task
>>>> ComputeManager.publish_service_capabilities from (pid=9785) periodic_tasks
>>>> /opt/stack/nova/nova/manager.py:171
>>>> 2012-11-21 22:46:56 DEBUG nova.manager [-] Notifying Schedulers of
>>>> capabilities ... from (pid=9785) publish_service_capabilities
>>>> /opt/stack/nova/nova/manager.py:256
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>> --
>>> Mohammed Naser — vexxhost
>>> -----------------------------------------------------
>>> D. 514-316-8872
>>> D. 800-910-1726 ext. 200
>>> E. mnaser at vexxhost.com
>>> W. http://vexxhost.com
>>>
>>>
>>
>>
>> --
>> Afef MDHAFFAR
>> http://www.redcad.org/members/mdhaffar/
>>
>>
>>
>>
>
>
> --
> Mohammed Naser — vexxhost
> -----------------------------------------------------
> D. 514-316-8872
> D. 800-910-1726 ext. 200
> E. mnaser at vexxhost.com
> W. http://vexxhost.com
>
>
--
Afef MDHAFFAR
http://www.redcad.org/members/mdhaffar/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20121122/d5205bc0/attachment.html>
More information about the Openstack
mailing list