[nova] instance not recoverable when its image backend got deleted
For an Openstack VM instance, if its backing image got deleted already, and this VM instance was already hard rebooted, and went into `Error` state due to `Image not found` error. Is it possible to recover this VM stucking in this bad state? I tried to create a vm snapshot, and then try to rebuild this VM using the newly created vm snapshot, and it kept failing. Regards, Hai
On Fri, 2023-02-03 at 15:16 -0600, hai wu wrote:
For an Openstack VM instance, if its backing image got deleted already, and this VM instance was already hard rebooted, and went into `Error` state due to `Image not found` error. a hard reboot does not try to download the image or even interact with glance if the instance went into error that likely means something esle happend that caused that.
in general deleteing an image form glance will have no impact on a running instnace and hard reboots or live migration will continue to work. evacuate or rebuild/rescue (without sepcifying a new image) will fail unless the image was boot form volume. the fact the image is deleted in glance is proably not the root cause of the issue but it will limit what you can do to recover. if the iamge is in use by a vm on the host and you are not using images_type=flat (i.e. you are using images_type=qcow or raw) the image shoudl still be in the compute nodes image cache. as an admin you could undelete the iamge in database and reupload it. if the vm has a disk that refecenes a backing file and that file nolonger exist on the compute host or glance there is not way to fully recover the vm you might be able to scp the file somewere and manualy extract teh vms data form the delta disk but there is nothign you can do form the openstack api to fix this.
Is it possible to recover this VM stucking in this bad state? I tried to create a vm snapshot, and then try to rebuild this VM using the newly created vm snapshot, and it kept failing.
Regards, Hai
On Fri, 2023-02-03 at 15:16 -0600, hai wu wrote:
For an Openstack VM instance, if its backing image got deleted already, and this VM instance was already hard rebooted, and went into `Error` state due to `Image not found` error. a hard reboot does not try to download the image or even interact with glance if the instance went into error that likely means something esle happend that caused that.
in general deleteing an image form glance will have no impact on a running instnace and hard reboots or live migration will continue to work. evacuate or rebuild/rescue (without sepcifying a new image) will fail unless the image was boot form volume. the fact the image is deleted in glance is proably not the root cause of the issue but it will limit what you can do to recover. if the iamge is in use by a vm on the host and you are not using images_type=flat (i.e. you are using images_type=qcow or raw) the image shoudl still be in the compute nodes image cache. as an admin you could undelete the iamge in database and reupload it. if the vm has a disk that refecenes a backing file and that file nolonger exist on the compute host or glance there is not way to fully recover the vm you might be able to scp the file somewere and manualy extract teh vms data form the delta disk but there is nothign you can do form the openstack api to fix this.
Is it possible to recover this VM stucking in this bad state? I tried to create a vm snapshot, and then try to rebuild this VM using the newly created vm snapshot, and it kept failing.
Regards, Hai
participants (2)
-
hai wu
-
Sean Mooney