[Openstack] Instances fail during/after networking
Antonio Messina
antonio.s.messina at gmail.com
Fri Nov 1 17:16:06 UTC 2013
On Fri, Nov 1, 2013 at 5:45 PM, Dheerendra
<dheerendra.madhusudhana at gmail.com> wrote:
> Hi Antonio
>
> I suspect the problem with rabbitMQ. Can you check it is up and running ?
RabbitMQ is up&running, but how can I check if it is *suffering*? The
daemon is not using that much CPU, while nova-network is around
80-100% of the cpu when I start an instance
I already increased the rpc_response_timeout value to 180 some times
ago because it was failing when trying to start more than 15 instances
at the same time, and it was working fine until a few days ago.
> Can you look at the controller logs as well ?
>From the server logs (in debug mode) I can't see anything if there is
anything wrong. I don't see any "ERROR", and I see a lot of amqp
messages that look all the same... I can provide the full logs if
someone is willing to take a look at it :)
.a.
>
> -Dheerendra
>
>
> On Thu, Oct 31, 2013 at 10:58 PM, Antonio Messina
> <antonio.s.messina at gmail.com> wrote:
>>
>> Hi All,
>>
>> We are experiencing a problem with our OpenStack Folsom installation
>> on Ubuntu 12.04, using nova-network: instances are NOT created, and
>> deleting instances in error state fails.
>>
>> First of all: the installation was in "production" for almost one
>> year, and we didn't change anything. The only change made was updating
>> some packages from ubuntu and from ubuntu-cloud archives.
>>
>> (PS: The following text is formatted as restructured text)
>>
>> In details what happen is:
>>
>> * The instance is created `Thu Oct 31 18:01:00 CET 2013`::
>>
>> 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi
>> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce] GET
>> http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
>> 2013-10-31 18:01:02 INFO nova.api.openstack.wsgi
>> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce]
>> http://cloud.gc3.uzh.ch:8774/v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
>> returned with HTTP 200
>> 2013-10-31 18:01:02 INFO nova.osapi_compute.wsgi.server
>> [req-a199cdd4-6f38-4767-b4c9-a9e40a2d1d23 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce] 130.60.24.9 - - [31/Oct/2013 18:01:02]
>> "GET
>> /v2/0b41ff4c9eec47ac9ba7ee2680a693ce/servers/ece6db13-10ee-4808-9646-57ab285ca33d
>> HTTP/1.1" 200 1531 0.225107
>>
>> * On the compute node, I see multiple times the following trace::
>>
>> 2013-10-31 18:01:26 931 INFO nova.compute.resource_tracker [-]
>> Compute_service record updated for node-09-01-00
>> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free
>> ram (MB): 17389
>> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free
>> disk (GB): 136
>> 2013-10-31 18:02:33 931 AUDIT nova.compute.resource_tracker [-] Free
>> VCPUS: 0
>> 2013-10-31 18:02:34 931 INFO nova.compute.resource_tracker [-]
>> Compute_service record updated for node-09-01-00
>> 2013-10-31 18:02:41 931 ERROR nova.openstack.common.rpc.amqp [-]
>> Exception during message handling
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> Traceback (most recent call last):
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 276, in _process_data
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp rval =
>> self.proxy.dispatch(ctxt, version, method, **args)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>> line 145, in dispatch
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
>> getattr(proxyobj, method)(ctxt, **kwargs)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> temp_level, payload)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
>> f(*args, **kw)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 390, in
>> refresh_instance_security_rules
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp return
>> self.driver.refresh_instance_security_rules(instance)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2243,
>> in refresh_instance_security_rules
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> self.firewall_driver.refresh_instance_security_rules(instance)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 446, in
>> refresh_instance_security_rules
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> self.do_refresh_instance_rules(instance)
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 463, in
>> do_refresh_instance_rules
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> network_info = self.network_infos[instance['id']]
>> 2013-10-31 18:02:41 931 TRACE nova.openstack.common.rpc.amqp
>> KeyError: 38331
>>
>> This backtrace does not seem to be relevant, as I've already seen
>> it in log of instances that started without problems.
>>
>> * After a while, IPs are assigned to the instance. I don't see
>> anything in the ``nova-network.log``, but I can see the IPs in the
>> ``nova list`` output. I also see the following errors in the compute
>> node::
>>
>> 2013-10-31 18:04:04 931 ERROR nova.openstack.common.rpc.common [-]
>> Timed out waiting for RPC response: timed out
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> Traceback (most recent call last):
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 518, in ensure
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return method(*args, **kwargs)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 596, in _consume
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return self.connection.drain_events(timeout=timeout)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in
>> drain_events
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return self.transport.drain_events(self.connection, **kwargs)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238,
>> in drain_events
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return connection.drain_events(**kwargs)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in
>> drain_events
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return self.wait_multi(self.channels.values(), timeout=timeout)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in
>> wait_multi
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> chanmap.keys(), allowed_methods, timeout=timeout)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120,
>> in _wait_multiple
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> channel, method_sig, args, content = read_timeout(timeout)
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in
>> read_timeout
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> return self.method_reader.read_method()
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common File
>> "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>> line 221, in read_method
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> raise m
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> timeout: timed out
>> 2013-10-31 18:04:04 931 TRACE nova.openstack.common.rpc.common
>> 2013-10-31 18:04:04 ERROR nova.compute.manager
>> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] Instance failed network setup
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] Traceback (most recent call last):
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in
>> _allocate_network
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] requested_networks=requested_networks)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in wrapper
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] res = f(self, context, *args,
>> **kwargs)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in
>> allocate_for_instance
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] 'args': args})
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
>> line 108, in call
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] return _get_impl().call(cfg.CONF,
>> context, topic, msg, timeout)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 718, in call
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] rpc_amqp.get_connection_pool(conf,
>> Connection))
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 369, in call
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] rv = list(rv)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 330, in __iter__
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] self.done()
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] self.gen.next()
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 327, in __iter__
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] self._iterator.next()
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 601, in iterconsume
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] yield self.ensure(_error_callback,
>> _consume)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 521, in ensure
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] error_callback(e)
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 582, in _error_callback
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] raise rpc_common.Timeout()
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] Timeout: Timeout while waiting on RPC
>> response.
>> 2013-10-31 18:04:04 931 TRACE nova.compute.manager [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d]
>> 2013-10-31 18:04:04 INFO nova.compute.resource_tracker
>> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce] Aborting claim: [Claim
>> ece6db13-10ee-4808-9646-57ab285ca33d: 14336 MB memory, 100 GB disk, 8 VCPUS]
>>
>> and then again::
>>
>> 2013-10-31 18:07:04 ERROR nova.compute.manager
>> [req-bf9a591c-7e6e-4d7e-92db-f08f92bb373b 285413c4eae74545bbd066b04af2ffd4
>> 0b41ff4c9eec47ac9ba7ee2680a693ce] [instance:
>> ece6db13-10ee-4808-9646-57ab285ca33d] Build error: ['Traceback (most recent
>> call last):\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 498, in
>> _run_instance\n requested_networks)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 721, in
>> _allocate_network\n requested_networks=requested_networks)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 44, in
>> wrapper\n res = f(self, context, *args, **kwargs)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 249, in
>> allocate_for_instance\n \'args\': args})\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
>> line 108, in call\n return _get_impl().call(cfg.CONF, context, topic,
>> msg, timeout)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 718, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', '
>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>> line 369, in call\n rv = list(rv)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 330, in __iter__\n self.done()\n', ' File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>> self.gen.next()\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 327, in __iter__\n self._iterator.next()\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 601, in iterconsume\n yield self.ensure(_error_callback,
>> _consume)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 521, in ensure\n error_callback(e)\n', ' File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 582, in _error_callback\n raise rpc_common.Timeout()\n', 'Timeout:
>> Timeout while waiting on RPC response.\n']
>> 2013-10-31 18:07:05 931 ERROR nova.openstack.common.rpc.amqp [-]
>> Exception during message handling
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> Traceback (most recent call last):
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 276, in _process_data
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rval =
>> self.proxy.dispatch(ctxt, version, method, **args)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>> line 145, in dispatch
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
>> getattr(proxyobj, method)(ctxt, **kwargs)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> temp_level, payload)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
>> f(*args, **kw)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in
>> decorated_function
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> pass
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in
>> decorated_function
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
>> function(self, context, *args, **kwargs)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in
>> decorated_function
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> kwargs['instance']['uuid'], e, sys.exc_info())
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in
>> decorated_function
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
>> function(self, context, *args, **kwargs)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in
>> run_instance
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> do_run_instance()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp retval
>> = f(*args, **kwargs)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in
>> do_run_instance
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> admin_password, is_first_time, instance)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in
>> _run_instance
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self._set_instance_error_state(context, instance['uuid'])
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in
>> _run_instance
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> is_first_time, request_spec, filter_properties)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 547, in
>> _reschedule_or_reraise
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self._deallocate_network(context, instance)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 783, in
>> _deallocate_network
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.network_api.deallocate_for_instance(context, instance)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 261, in
>> deallocate_for_instance
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp 'args':
>> args})
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
>> line 108, in call
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp return
>> _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 718, in call
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> rpc_amqp.get_connection_pool(conf, Connection))
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 369, in call
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp rv =
>> list(rv)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 330, in __iter__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.done()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 327, in __iter__
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> self._iterator.next()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 601, in iterconsume
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp yield
>> self.ensure(_error_callback, _consume)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 521, in ensure
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> error_callback(e)
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
>> line 582, in _error_callback
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp raise
>> rpc_common.Timeout()
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>> Timeout: Timeout while waiting on RPC response.
>> 2013-10-31 18:07:05 931 TRACE nova.openstack.common.rpc.amqp
>>
>> * After this, the instance is in ERROR state. If I delete it, it
>> usually only release the IPs, and only after a while, if I re-run
>> ``nova delete``, it *sometimes* actually delete the instance.
>>
>> This is driving me crazy, I don't really know where to look for
>> hints. I guess the problem is in the network node, but I can't
>> understand anything from the logs.
>>
>> can someone help me with this?
>>
>> Thank you in advance
>>
>> .a.
>>
>> --
>> antonio.s.messina at gmail.com
>> antonio.messina at uzh.ch +41 (0)44 635 42 22
>> GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/
>> University of Zurich
>> Winterthurerstrasse 190
>> CH-8057 Zurich Switzerland
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
--
antonio.s.messina at gmail.com
antonio.messina at uzh.ch +41 (0)44 635 42 22
GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
More information about the Openstack
mailing list