Hello, Did you check /var/log/libvirt/libvirtd.log? Cheers, -- Arnaud Morin On 03.09.20 - 10:59, Grant Morley wrote:

Hi All,

I was wondering if anyone has come across an issue with libvirt seemingly having an issue with instances all of a sudden "locking up" with the following error:

Failed to terminate process 2263874 with SIGKILL: Device or resource busy

In the nova logs I am seeing:

2020-09-03 09:13:43.208 2659995 INFO nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Rebooting instance 2020-09-03 09:15:52.429 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Failed to soft reboot instance. Trying hard reboot. 2020-09-03 09:16:32.450 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 1 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:12.484 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 2 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:52.516 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 3 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:52.526 2659995 ERROR nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Cannot reboot instance: Failed to terminate process 2263874 with SIGKILL: Device or resource busy: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:53.026 2659995 INFO nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Successfully reverted task state from reboot_started on failure for instance.

It seems to be caused when a reboot happens to an instance.

If you reset the state and try again, the same error occurs.  You also seemingly cannot kill off any libvirt process that is attached to that instance.

To me it looks like it could be a kernel issue with libvirt but I could be wrong?

Does anyone know of a workaround for this other than maybe restarting a compute host?

Many thanks,