Open Stack

On Sep 24, 2012, at 8:06 AM, Salman Toor <salman.toor at it.uu.se> wrote:

> Hi,
>
>
> I want to start an instance on a newly deployed compute node. But got following error messages.
>
>
> ------------ host: openstack-com, compute.log ----------------------------
>
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=5116) _publish_service_capabilities /usr/lib/python2.6/site-packages/nova/manager.py:203
> 2012-09-24 13:43:07 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=5116) fanout_cast /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:359
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._sync_power_states from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 INFO nova.virt.libvirt.connection [-] Compute_service record updated for openstack-com.cern.ch
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 18 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 38 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=5116) _reclaim_queued_deletes /usr/lib/python2.6/site-packages/nova/compute/manager.py:2404
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:43:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=5116) _publish_service_capabilities /usr/lib/python2.6/site-packages/nova/manager.py:203
> 2012-09-24 13:44:07 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=5116) fanout_cast /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:359
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 10 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:07 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:08 INFO nova.virt.libvirt.connection [-] Compute_service record updated for openstack-com.cern.ch
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 17 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 37 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:08 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=5116) _reclaim_queued_deletes /usr/lib/python2.6/site-packages/nova/compute/manager.py:2404
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:08 INFO nova.compute.manager [-] Updating host status
> 2012-09-24 13:44:08 DEBUG nova.virt.libvirt.connection [-] Updating host stats from (pid=5116) update_status /usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py:2500
> 2012-09-24 13:44:08 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:44:18 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c', u'_context_read_deleted': u'no', u'args': {u'instance_uuid': u'47c3e22d-75f1-4fd3-95aa-2fa3db0f666a', u'requested_networks': None, u'is_first_time': True, u'admin_password': '<SANITIZED>', u'injected_files': []}, u'_context_auth_token': '<SANITIZED>', u'_context_is_admin': True, u'_context_project_id': u'461226cb29194c86b84964896e2fefbc', u'_context_timestamp': u'2012-09-24T11:44:18.004777', u'_context_user_id': u'91fa17b603c74c238e89d3b334bb0ceb', u'method': u'run_instance', u'_context_remote_address': u'188.184.20.238'} from (pid=5116) _safe_log /usr/lib/python2.6/site-packages/nova/rpc/common.py:160
> 2012-09-24 13:44:18 DEBUG nova.rpc.amqp [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] unpacked context: {'user_id': u'91fa17b603c74c238e89d3b334bb0ceb', 'roles': [u'admin'], 'timestamp': '2012-09-24T11:44:18.004777', 'auth_token': '<SANITIZED>', 'remote_address': u'188.184.20.238', 'is_admin': True, 'request_id': u'req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c', 'project_id': u'461226cb29194c86b84964896e2fefbc', 'read_deleted': u'no'} from (pid=5116) _safe_log /usr/lib/python2.6/site-packages/nova/rpc/common.py:160
> 2012-09-24 13:44:18 DEBUG nova.utils [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] Attempting to grab semaphore "47c3e22d-75f1-4fd3-95aa-2fa3db0f666a" for method "do_run_instance"... from (pid=5116) inner /usr/lib/python2.6/site-packages/nova/utils.py:928
> 2012-09-24 13:44:18 DEBUG nova.utils [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] Got semaphore "47c3e22d-75f1-4fd3-95aa-2fa3db0f666a" for method "do_run_instance"... from (pid=5116) inner /usr/lib/python2.6/site-packages/nova/utils.py:932
> 2012-09-24 13:44:18 AUDIT nova.compute.manager [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a] Starting instance...
> 2012-09-24 13:44:19 DEBUG nova.rpc.amqp [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] Making asynchronous call on network ... from (pid=5116) multicall /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:326
> 2012-09-24 13:44:19 DEBUG nova.rpc.amqp [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] MSG_ID is 55e70c2bf3084c4b880b025130e33295 from (pid=5116) multicall /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:329
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=5116) _publish_service_capabilities /usr/lib/python2.6/site-packages/nova/manager.py:203
> 2012-09-24 13:45:08 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=5116) fanout_cast /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:359
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 9 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:08 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:09 INFO nova.virt.libvirt.connection [-] Compute_service record updated for openstack-com.cern.ch
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 16 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:09 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=5116) multicall /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:326
> 2012-09-24 13:45:09 DEBUG nova.rpc.amqp [-] MSG_ID is 3702e883e2e8421f839332004922333f from (pid=5116) multicall /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:329
> 2012-09-24 13:45:09 DEBUG nova.compute.manager [-] Updated the info_cache for instance 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a from (pid=5116) _heal_instance_info_cache /usr/lib/python2.6/site-packages/nova/compute/manager.py:2251
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 36 ticks left until next run from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:147
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:09 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=5116) _reclaim_queued_deletes /usr/lib/python2.6/site-packages/nova/compute/manager.py:2404
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:09 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=5116) periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:152
> 2012-09-24 13:45:19 ERROR nova.rpc.impl_qpid [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] Timed out waiting for RPC response: None
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid Traceback (most recent call last):
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 364, in ensure
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid     return method(*args, **kwargs)
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 413, in _consume
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid   File "<string>", line 6, in next_receiver
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 651, in next_receiver
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid     raise Empty
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid Empty: None
> 2012-09-24 13:45:19 TRACE nova.rpc.impl_qpid
> 2012-09-24 13:45:19 ERROR nova.compute.manager [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a] Instance failed network setup
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a] Traceback (most recent call last):
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 578, in _allocate_network
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     requested_networks=requested_networks)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 178, in allocate_for_instance
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     'args': args})
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     return _get_impl().call(context, topic, msg, timeout)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 526, in call
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 343, in call
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     rv = list(rv)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 304, in __iter__
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     self.done()
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     self.gen.next()
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 301, in __iter__
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     self._iterator.next()
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 422, in iterconsume
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     yield self.ensure(_error_callback, _consume)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 368, in ensure
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     error_callback(e)
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 407, in _error_callback
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]     raise rpc_common.Timeout()
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a] Timeout: Timeout while waiting on RPC response.
> 2012-09-24 13:45:19 TRACE nova.compute.manager [instance: 47c3e22d-75f1-4fd3-95aa-2fa3db0f666a]
> 2012-09-24 13:45:19 ERROR nova.rpc.amqp [req-7a85d8e9-73d2-4b99-aa52-5544ac29b96c 91fa17b603c74c238e89d3b334bb0ceb 461226cb29194c86b84964896e2fefbc] Exception during message handling
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 253, in _process_data
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     rval = node_func(context=ctxt, **node_args)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     return f(*args, **kw)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 183, in decorated_function
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     sys.exc_info())
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self.gen.next()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     return function(self, context, instance_uuid, *args, **kwargs)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 675, in run_instance
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     do_run_instance()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 946, in inner
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 674, in do_run_instance
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self._run_instance(context, instance_uuid, **kwargs)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 470, in _run_instance
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self._set_instance_error_state(context, instance_uuid)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self.gen.next()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 443, in _run_instance
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     requested_networks)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 578, in _allocate_network
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     requested_networks=requested_networks)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 178, in allocate_for_instance
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     'args': args})
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     return _get_impl().call(context, topic, msg, timeout)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 526, in call
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 343, in call
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     rv = list(rv)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 304, in __iter__
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self.done()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self.gen.next()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 301, in __iter__
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     self._iterator.next()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 422, in iterconsume
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     yield self.ensure(_error_callback, _consume)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 368, in ensure
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     error_callback(e)
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 407, in _error_callback
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp     raise rpc_common.Timeout()
> 2012-09-24 13:45:19 TRACE nova.rpc.amqp Timeout: Timeout while waiting on RPC response.
>
> service status:
>
> Binary           Host                                 Zone             Status     State Updated_At
> nova-compute     openstack-con.cern.ch                nova             enabled    XXX   2012-09-24 09:35:59
> nova-network     openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:23
> nova-scheduler   openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:21
> nova-volume      openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:16
> nova-cert        openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:20
> nova-consoleauth openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:16
> nova-compute     openstack-com.cern.ch                nova             enabled    :-)   2012-09-24 11:47:22
> nova-cert        openstack-com.cern.ch                nova             enabled    :-)   2012-09-24 11:47:23
> nova-console     openstack-con.cern.ch                nova             enabled    :-)   2012-09-24 11:47:21
>
>
> in order to clear the test deployment
>
> 1 - openstack-con, Running as a controller node. running network, compute. glance, keystone, volume, api, cert, consoleauth, objectstore, libvertd and qpidd
>
> 2 - openstack-com, Running as a Compute node. api, cert, compute and libvertd
>
> Do I really need to run api and cert on each compute node?
>
> It is clear that its something related to messaging. But if the problem in communication than how come openstack-com communicates with openstack-con network service.....
>
> qpidd.conf
> ----
> cluster-mechanism=DIGEST-MD5 ANONYMOUS
> auth=no
> ----
>
> Please give me some hint.
>
> Regards..
> Salman.
>
>
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


I have seen this happen on 12.04. RabbitMQ seems to be somewhat
unstable at times with high volume.