Issue with libvirt unable to kill processes
Grant Morley
grant at civo.com
Thu Sep 3 09:59:25 UTC 2020
Hi All,
I was wondering if anyone has come across an issue with libvirt
seemingly having an issue with instances all of a sudden "locking up"
with the following error:
Failed to terminate process 2263874 with SIGKILL: Device or resource busy
In the nova logs I am seeing:
2020-09-03 09:13:43.208 2659995 INFO nova.compute.manager
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a]
Rebooting instance
2020-09-03 09:15:52.429 2659995 WARNING nova.virt.libvirt.driver
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Failed
to soft reboot instance. Trying hard reboot.
2020-09-03 09:16:32.450 2659995 WARNING nova.virt.libvirt.driver
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error
from libvirt during destroy. Code=38 Error=Failed to terminate process
2263874 with SIGKILL: Device or resource busy; attempt 1 of 3:
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device
or resource busy
2020-09-03 09:17:12.484 2659995 WARNING nova.virt.libvirt.driver
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error
from libvirt during destroy. Code=38 Error=Failed to terminate process
2263874 with SIGKILL: Device or resource busy; attempt 2 of 3:
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device
or resource busy
2020-09-03 09:17:52.516 2659995 WARNING nova.virt.libvirt.driver
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error
from libvirt during destroy. Code=38 Error=Failed to terminate process
2263874 with SIGKILL: Device or resource busy; attempt 3 of 3:
libvirtError: Failed to terminate process 2263874 with SIGKILL: Device
or resource busy
2020-09-03 09:17:52.526 2659995 ERROR nova.compute.manager
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Cannot
reboot instance: Failed to terminate process 2263874 with SIGKILL:
Device or resource busy: libvirtError: Failed to terminate process
2263874 with SIGKILL: Device or resource busy
2020-09-03 09:17:53.026 2659995 INFO nova.compute.manager
[req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc
6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf -
default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a]
Successfully reverted task state from reboot_started on failure for
instance.
It seems to be caused when a reboot happens to an instance.
If you reset the state and try again, the same error occurs. You also
seemingly cannot kill off any libvirt process that is attached to that
instance.
To me it looks like it could be a kernel issue with libvirt but I could
be wrong?
Does anyone know of a workaround for this other than maybe restarting a
compute host?
Many thanks,
More information about the openstack-discuss
mailing list