[Openstack] Instances fail during/after networking

Antonio Messina antonio.s.messina at gmail.com
Fri Nov 1 17:19:46 UTC 2013


Yesterday we tried to shut down all the services on the compute nodes
and the controller node (including mysql and rabbitmq). For a little
while everything seemed working fine, we were able to start a few
instances very rapidly and without problems. After a while, however,
the problem came up again:

* if the instance starts, it takes a lot of times (minutes)
* 2 over 3 instances goes in error state
* deleting instances takes *ages*

Anyone willing to help me debugging this thing?

.a.

On Thu, Oct 31, 2013 at 6:28 PM, Antonio Messina
<antonio.s.messina at gmail.com> wrote:
> Hi All,
>
> We are experiencing a problem with our OpenStack Folsom installation
> on Ubuntu 12.04, using nova-network: instances are NOT created, and
> deleting instances in error state fails.
>
> First of all: the installation was in "production" for almost one
> year, and we didn't change anything. The only change made was updating
> some packages from ubuntu and from ubuntu-cloud archives.
>
> (PS: The following text is formatted as restructured text)
>
> In details what happen is:
>
> * The instance is created `Thu Oct 31 18:01:00 CET 2013`::
>
>       2013-10-31 18:01:02 INFO nova.api.openstack.wsgi [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] GET http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
>       2013-10-31 18:01:02 INFO nova.api.openstack.wsgi [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d returned with HTTP 200
>       2013-10-31 18:01:02 INFO nova.osapi_compute.wsgi.server [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] 130.60.24.9 - - [31/Oct/2013 18:01:02] "GET /v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d HTTP/1.1" 200 1531 0.225107
>
> * On the compute node, I see multiple times the following trace::
>
>       2013-10-31 18:01:26 931 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-09-01-00
>       2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 17389
>       2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 136
>       2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 0
>       2013-10-31 18:02:34 931 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-09-01-00
>       2013-10-31 18:02:41 931 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 390, in refresh_instance_security_rules
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return self.driver.refresh_instance_security_rules(instance)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2243, in refresh_instance_security_rules
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp self.firewall_driver.refresh_instance_security_rules(instance)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 446, in refresh_instance_security_rules
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp self.do_refresh_instance_rules(instance)
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 463, in do_refresh_instance_rules
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp network_info = self.network_infos[instance['id']]
>       2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp KeyError: 38331
>
>   This backtrace does not seem to be relevant, as I've already seen
>   it in log of instances that started without problems.
>
> * After a while, IPs are assigned to the instance. I don't see
>   anything in the ``nova-network.log``, but I can see the IPs in the
>   ``nova list`` output. I also see the following errors in the compute
>   node::
>
>       2013-10-31 18:04:04 931 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 518, in ensure
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return method(*args, **kwargs)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 596, in _consume
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return self.connection.drain_events(timeout=timeout)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return self.transport.drain_events(self.connection, **kwargs)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return connection.drain_events(**kwargs)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return self.wait_multi(self.channels.values(), timeout=timeout)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  chanmap.keys(), allowed_methods, timeout=timeout)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  channel, method_sig, args, content = read_timeout(timeout)
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common  return self.method_reader.read_method()
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common     raise m
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common timeout: timed out
>       2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>       2013-10-31 18:04:04 ERROR nova.compute.manager [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: ece6db13-10ee-4808-9646-57ab285ca33d] Instance failed network setup
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d] Traceback (most recent call last):
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in _allocate_network
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d] requested_networks=requested_networks)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     res = f(self, context, *args, **kwargs)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in allocate_for_instance
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     'args': args})
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 718, in call
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d] rpc_amqp.get_connection_pool(conf, Connection))
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     rv = list(rv)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     self.done()
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     self.gen.next()
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     self._iterator.next()
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 601, in iterconsume
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     yield self.ensure(_error_callback, _consume)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 521, in ensure
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     error_callback(e)
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 582, in _error_callback
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]     raise rpc_common.Timeout()
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d] Timeout: Timeout while waiting on RPC response.
>       2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance: ece6db13-10ee-4808-9646-57ab285ca33d]
>       2013-10-31 18:04:04 INFO nova.compute.resource_tracker [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] Aborting claim: [Claim ece6db13-10ee-4808-9646-57ab285ca33d: 14336 MB memory, 100 GB disk, 8 VCPUS]
>
>   and then again::
>
>       2013-10-31 18:07:04 ERROR nova.compute.manager [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance: ece6db13-10ee-4808-9646-57ab285ca33d] Build error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 498, in _run_instance\n    requested_networks)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in _allocate_network\n    requested_networks=requested_networks)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper\n    res = f(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in allocate_for_instance\n    \'args\': args})\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 108, in call\n    return _get_impl().call(cfg.CONF, context, topic, msg, timeout)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 718, in call\n    rpc_amqp.get_connection_pool(conf, Connection))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369, in call\n    rv = list(rv)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__\n    self.done()\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__\n    self._iterator.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 601, in iterconsume\n    yield self.ensure(_error_callback, _consume)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 521, in ensure\n    error_callback(e)\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 582, in _error_callback\n    raise rpc_common.Timeout()\n', 'Timeout: Timeout while waiting on RPC response.\n']
>       2013-10-31 18:07:05 931 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in decorated_function
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp     pass
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in decorated_function
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in decorated_function
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in decorated_function
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in run_instance
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp do_run_instance()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in do_run_instance
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in _run_instance
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in _run_instance
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp is_first_time, request_spec, filter_properties)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 547, in _reschedule_or_reraise
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self._deallocate_network(context, instance)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 783, in _deallocate_network
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.network_api.deallocate_for_instance(context, instance)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 261, in deallocate_for_instance
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp 'args': args})
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 718, in call
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.done()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self.gen.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp self._iterator.next()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 601, in iterconsume
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp yield self.ensure(_error_callback, _consume)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 521, in ensure
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp error_callback(e)
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 582, in _error_callback
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp raise rpc_common.Timeout()
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response.
>       2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>
> * After this, the instance is in ERROR state. If I delete it, it
>   usually only release the IPs, and only after a while, if I re-run
>   ``nova delete``, it *sometimes* actually delete the instance.
>
> This is driving me crazy, I don't really know where to look for
> hints. I guess the problem is in the network node, but I can't
> understand anything from the logs.
>
> can someone help me with this?
>
> Thank you in advance
>
> .a.
>
> --
> antonio.s.messina at gmail.com
> antonio.messina at uzh.ch                     +41 (0)44 635 42 22
> GC3: Grid Computing Competence Center      http://www.gc3.uzh.ch/
> University of Zurich
> Winterthurerstrasse 190
> CH-8057 Zurich Switzerland



-- 
antonio.s.messina at gmail.com
antonio.messina at uzh.ch                     +41 (0)44 635 42 22
GC3: Grid Computing Competence Center      http://www.gc3.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland




More information about the Openstack mailing list