[openstack-dev] [nova] nova-compute not re-establishing connectivity after controller switchover
Chris Friesen
chris.friesen at windriver.com
Mon Mar 24 16:41:15 UTC 2014
We've been stress-testing our system doing controlled switchover of the controller. Normally this works okay, but we've run into a situation that seems to show a flaw in the reconnection logic.
On the compute node, nova-compute has managed to get into a state where it shows as "down" in "nova service-list", and the nova-compute.log seems to show it never managing to reconnect with the AMQP server.
I've included logs below showing what seems to be the beginning of the problem and then showing it transitioning to the periodic logs without a successful reconnection. The periodic logs have now been going for roughly seven hours...
Any ideas on what might be going on would be appreciated.
Chris
2014-03-24 09:24:33.566 6620 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'sgw-4', vm_state=u'active', task_state=None, vcpus=2, cpuset=0x180, cpulist=[7, 8] pinned, nodelist=[0], node=0
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'sgw-1', vm_state=u'active', task_state=None, vcpus=2, cpuset=0x60, cpulist=[5, 6] pinned, nodelist=[0], node=0
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'load_balancer', vm_state=u'active', task_state=None, vcpus=3, cpuset=0x1c00, cpulist=[10, 11, 12] pinned, nodelist=[1], node=1
2014-03-24 09:24:34.182 6620 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 111290, per-node: [52286, 59304], numa nodes:2
2014-03-24 09:24:34.183 6620 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 29
2014-03-24 09:24:34.183 6620 AUDIT nova.compute.resource_tracker [-] Free vcpus: 170, free per-node float vcpus: [48, 112], free per-node pinned vcpus: [3, 7]
2014-03-24 09:24:34.183 6620 INFO nova.compute.resource_tracker [-] DETAIL: vcpus:20, Free vcpus:170, 16.0x overcommit, per-cpu float cpulist: [3, 4, 9, 13, 14, 15, 16, 17, 18, 19]
2014-03-24 09:24:34.244 6620 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute-0:compute-0
2014-03-24 09:25:36.564 6620 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'sgw-4', vm_state=u'active', task_state=None, vcpus=2, cpuset=0x180, cpulist=[7, 8] pinned, nodelist=[0], node=0
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'sgw-1', vm_state=u'active', task_state=None, vcpus=2, cpuset=0x60, cpulist=[5, 6] pinned, nodelist=[0], node=0
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL: instance: name=u'load_balancer', vm_state=u'active', task_state=None, vcpus=3, cpuset=0x1c00, cpulist=[10, 11, 12] pinned, nodelist=[1], node=1
2014-03-24 09:25:37.182 6620 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 111290, per-node: [52286, 59304], numa nodes:2
2014-03-24 09:25:37.182 6620 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 29
2014-03-24 09:25:37.183 6620 AUDIT nova.compute.resource_tracker [-] Free vcpus: 170, free per-node float vcpus: [48, 112], free per-node pinned vcpus: [3, 7]
2014-03-24 09:25:37.183 6620 INFO nova.compute.resource_tracker [-] DETAIL: vcpus:20, Free vcpus:170, 16.0x overcommit, per-cpu float cpulist: [3, 4, 9, 13, 14, 15, 16, 17, 18, 19]
2014-03-24 09:25:37.245 6620 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute-0:compute-0
2014-03-24 09:26:47.324 6620 ERROR root [-] Unexpected exception occurred 1 time(s)... retrying.
2014-03-24 09:26:47.324 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 745, in _consumer_thread
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 736, in consume
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 663, in iterconsume
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:26:47.324 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 658, in _consume
2014-03-24 09:26:47.324 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/connection.py", line 279, in drain_events
2014-03-24 09:26:47.324 6620 TRACE root return self.transport.drain_events(self.connection, **kwargs)
2014-03-24 09:26:47.324 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/transport/pyamqp.py", line 90, in drain_events
2014-03-24 09:26:47.324 6620 TRACE root return connection.drain_events(**kwargs)
2014-03-24 09:26:47.324 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 303, in drain_events
2014-03-24 09:26:47.324 6620 TRACE root return amqp_method(channel, args)
2014-03-24 09:26:47.324 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 509, in _close
2014-03-24 09:26:47.324 6620 TRACE root (class_id, method_id), ConnectionError)
2014-03-24 09:26:47.324 6620 TRACE root ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2014-03-24 09:26:47.324 6620 TRACE root
2014-03-24 09:26:47.366 6620 ERROR nova.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 104] Connection reset by peer
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 658, in _consume
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/connection.py", line 279, in drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/transport/pyamqp.py", line 90, in drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 303, in drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common return amqp_method(channel, args)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 506, in _close
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common self._x_close_ok()
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/connection.py", line 534, in _x_close_ok
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common self._send_method((10, 51))
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 62, in _send_method
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common self.channel_id, method_sig, args, content,
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 227, in write_method
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common write_frame(1, channel, payload)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/transport.py", line 183, in write_frame
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common frame_type, channel, size, payload, 0xce,
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common tail = self.send(data, flags)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common total_sent += fd.send(data[total_sent:], flags)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common error: [Errno 104] Connection reset by peer
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
2014-03-24 09:26:47.368 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:47.378 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-03-24 09:26:48.103 6620 ERROR nova.openstack.common.rpc.common [-] Failed to publish message to topic 'conductor': [Errno 104] Connection reset by peer
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 689, in _publish
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 369, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 316, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 324, in reconnect
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 82, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common self.revive(self._channel)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 216, in revive
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common self.declare()
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 102, in declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common self.exchange.declare()
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 166, in declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common nowait=nowait, passive=passive,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 604, in exchange_declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common self._send_method((40, 10), args)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 62, in _send_method
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common self.channel_id, method_sig, args, content,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 227, in write_method
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common write_frame(1, channel, payload)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/amqp/transport.py", line 183, in write_frame
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common frame_type, channel, size, payload, 0xce,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common tail = self.send(data, flags)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common total_sent += fd.send(data[total_sent:], flags)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common error: [Errno 104] Connection reset by peer
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
2014-03-24 09:26:48.105 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:48.112 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-03-24 09:26:48.327 6620 ERROR root [-] Unexpected exception occurred 1 time(s)... retrying.
2014-03-24 09:26:48.327 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 745, in _consumer_thread
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 736, in consume
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 663, in iterconsume
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 656, in _consume
2014-03-24 09:26:48.327 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 191, in consume
2014-03-24 09:26:48.327 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:26:48.327 6620 TRACE root nowait=nowait)
2014-03-24 09:26:48.327 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in basic_consume
2014-03-24 09:26:48.327 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:26:48.327 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-03-24 09:26:48.327 6620 TRACE root raise RecoverableConnectionError('connection already closed')
2014-03-24 09:26:48.327 6620 TRACE root RecoverableConnectionError: connection already closed
2014-03-24 09:26:48.327 6620 TRACE root
2014-03-24 09:26:48.379 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:48.386 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-03-24 09:26:49.112 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:49.120 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-03-24 09:26:51.386 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:51.402 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-03-24 09:26:52.120 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:52.127 6620 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-03-24 09:26:56.408 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:56.454 6620 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:57.128 6620 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:57.137 6620 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.2:5672
2014-03-24 09:27:39.807 6620 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._instance_usage_audit: Timeout while waiting on RPC response - topic: "conductor", RPC method: "task_log_get" info: "<unknown>"
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4623, in _instance_usage_audit
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/compute/utils.py", line 363, in has_audit_been_run
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/conductor/api.py", line 273, in task_log_get
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 403, in task_log_get
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC method: "task_log_get" info: "<unknown>"
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:27:39.809 6620 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-03-24 09:27:49.335 6620 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-03-24 09:27:49.335 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 745, in _consumer_thread
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 736, in consume
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 663, in iterconsume
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 656, in _consume
2014-03-24 09:27:49.335 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 191, in consume
2014-03-24 09:27:49.335 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:27:49.335 6620 TRACE root nowait=nowait)
2014-03-24 09:27:49.335 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in basic_consume
2014-03-24 09:27:49.335 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:27:49.335 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-03-24 09:27:49.335 6620 TRACE root raise RecoverableConnectionError('connection already closed')
2014-03-24 09:27:49.335 6620 TRACE root RecoverableConnectionError: connection already closed
2014-03-24 09:27:49.335 6620 TRACE root
2014-03-24 09:27:57.139 6620 ERROR nova.servicegroup.drivers.db [-] model server went away
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 98, in _report_state
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/conductor/api.py", line 269, in service_update
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 397, in service_update
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC method: "service_update" info: "<unknown>"
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db
2014-03-24 09:27:57.141 6620 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 59.038519 sec
2014-03-24 09:28:40.346 6620 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Timeout while waiting on RPC response - topic: "conductor", RPC method: "object_class_action" info: "<unknown>"
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 5024, in update_available_resource
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py", line 495, in update_available_resource
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 106, in wrapper
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 492, in object_class_action
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC method: "object_class_action" info: "<unknown>"
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:28:50.342 6620 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-03-24 09:28:50.342 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 745, in _consumer_thread
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 736, in consume
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 663, in iterconsume
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 656, in _consume
2014-03-24 09:28:50.342 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 191, in consume
2014-03-24 09:28:50.342 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:28:50.342 6620 TRACE root nowait=nowait)
2014-03-24 09:28:50.342 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in basic_consume
2014-03-24 09:28:50.342 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:28:50.342 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-03-24 09:28:50.342 6620 TRACE root raise RecoverableConnectionError('connection already closed')
2014-03-24 09:28:50.342 6620 TRACE root RecoverableConnectionError: connection already closed
2014-03-24 09:28:50.342 6620 TRACE root
2014-03-24 09:28:57.144 6620 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 50.00323 sec
2014-03-24 09:29:40.351 6620 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._run_pending_deletes: Timeout while waiting on RPC response - topic: "conductor", RPC method: "object_class_action" info: "<unknown>"
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 5239, in _run_pending_deletes
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 106, in wrapper
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 492, in object_class_action
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC method: "object_class_action" info: "<unknown>"
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:29:51.349 6620 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-03-24 09:29:51.349 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 745, in _consumer_thread
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 736, in consume
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 663, in iterconsume
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 578, in ensure
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 656, in _consume
2014-03-24 09:29:51.349 6620 TRACE root File "./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py", line 191, in consume
2014-03-24 09:29:51.349 6620 TRACE root File "/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:29:51.349 6620 TRACE root nowait=nowait)
2014-03-24 09:29:51.349 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in basic_consume
2014-03-24 09:29:51.349 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:29:51.349 6620 TRACE root File "/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-03-24 09:29:51.349 6620 TRACE root raise RecoverableConnectionError('connection already closed')
2014-03-24 09:29:51.349 6620 TRACE root RecoverableConnectionError: connection already closed
2014-03-24 09:29:51.349 6620 TRACE root
2014-03-24 09:29:57.147 6620 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 50.002704 sec
More information about the OpenStack-dev
mailing list