Open Stack

Hi All,

I was wondering if anyone has come across an issue with libvirt 
seemingly having an issue with instances all of a sudden "locking up" 
with the following error:

Failed to terminate process 2263874 with SIGKILL: Device or resource busy

In the nova logs I am seeing:

2020-09-03 09:13:43.208 2659995 INFO nova.compute.manager 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] 
Rebooting instance
2020-09-03 09:15:52.429 2659995 WARNING nova.virt.libvirt.driver 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Failed 
to soft reboot instance. Trying hard reboot.
2020-09-03 09:16:32.450 2659995 WARNING nova.virt.libvirt.driver 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error 
from libvirt during destroy. Code=38 Error=Failed to terminate process 
2263874 with SIGKILL: Device or resource busy; attempt 1 of 3: 
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device 
or resource busy
2020-09-03 09:17:12.484 2659995 WARNING nova.virt.libvirt.driver 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error 
from libvirt during destroy. Code=38 Error=Failed to terminate process 
2263874 with SIGKILL: Device or resource busy; attempt 2 of 3: 
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device 
or resource busy
2020-09-03 09:17:52.516 2659995 WARNING nova.virt.libvirt.driver 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error 
from libvirt during destroy. Code=38 Error=Failed to terminate process 
2263874 with SIGKILL: Device or resource busy; attempt 3 of 3: 
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device 
or resource busy
2020-09-03 09:17:52.526 2659995 ERROR nova.compute.manager 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Cannot 
reboot instance: Failed to terminate process 2263874 with SIGKILL: 
Device or resource busy: libvirtError: Failed to terminate process 
2263874 with SIGKILL: Device or resource busy
2020-09-03 09:17:53.026 2659995 INFO nova.compute.manager 
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - 
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] 
Successfully reverted task state from reboot_started on failure for 
instance.

It seems to be caused when a reboot happens to an instance.

If you reset the state and try again, the same error occurs.  You also 
seemingly cannot kill off any libvirt process that is attached to that 
instance.

To me it looks like it could be a kernel issue with libvirt but I could 
be wrong?

Does anyone know of a workaround for this other than maybe restarting a 
compute host?

Many thanks,