Open Stack

Looks like one or more of your VMs get stuck and libvirt having problems to stop it now: 

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher libvirtError: Failed to terminate process 8756 with SIGKILL: Device or resource busy


I would recommend you to check what exactly is PID (8756) in the error? I assume is a qemu process and what's going on with this process? 
Does it have some issue with accessing storage or something else? You can also check the libvirtd logs, there should be something more.

====================================================
Radosław Śmigielski
ALCATEL-LUCENT - Cloudband/Node Team
Office: 238 646 13 or +353 (0)188 646 13
Mobile: +353 86 820 74 23




________________________________________
From: cosmos cosmos [cosmos0703 at gmail.com]
Sent: 22 May 2014 12:30
To: openstack-dev at lists.openstack.org
Subject: [openstack-dev]  [sahara] Deleting cluster in sahara horizon, 500 Error occured in nova.

Hello My name is inhye park from samsung SDS.

I am a developer about nova and sahara.

Recently Our system updated to icehouse.

After deleting cluster in sahara, VM occurred 500 error.

But this problem not occur in nova cli command.

Only in sahara this error occurred.

This is error log.

-----------------------------------------------------------------------------------------

2014-05-22 20:06:42.631 25640 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Failed to terminate process 8756 with SIGKILL: Device or resource busy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     incoming.message))

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     payload)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     function(self, context, *args, **kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 296, in decorated_function

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2262, in terminate_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     do_terminate_instance(instance, bdms)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2260, in do_terminate_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     self._set_instance_error_state(context, instance['uuid'])

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2250, in do_terminate_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     reservations=reservations)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 103, in inner

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     rv = f(*args, **kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2220, in _delete_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     user_id=user_id)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2190, in _delete_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     self._shutdown_instance(context, db_inst, bdms)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _shutdown_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     requested_networks)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2115, in _shutdown_instance

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     block_device_info)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 951, in destroy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     self._destroy(instance)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 908, in _destroy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     instance=instance)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 880, in _destroy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     virt_dom.destroy()

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     rv = execute(f,*args,**kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     rv = meth(*args,**kwargs)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher libvirtError: Failed to terminate process 8756 with SIGKILL: Device or resource busy

2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher

2014-05-22 20:06:42.633 25640 ERROR oslo.messaging._drivers.common [-] Returning exception Failed to terminate process 8756 with SIGKILL: Device or resource busy to caller


Could you please help about this case?

Thanks.