[Openstack] Recovering VMs in OPenStack ESSEX

Razique Mahroua razique.mahroua at gmail.com
Wed Nov 13 06:20:23 UTC 2013


Regarding your last question. The boot error mostly happen when the disk 
has been modified (let's say power outage or shared-disk disconnect) 
while it was running, messing with the boot sectors.
You can use the nova repair command to repair an instance, or you make 
snapshots regularly. Other option is to plug a recovery disk into the 
domain (using
> virsh disk-attach

You can use the appropriate recovery systems depending on your images 
(GParted, DART, Ubuntu in recovery mode, etc...)

On 12 Nov 2013, at 0:41, Razique Mahroua wrote:

> Ok,
> in that case, use kpartx from the host against the disk image to 
> discover partitions:
> kpartx -lv /var/lib/nova/[…]/disk
> to see if you find any partition
>
> You can as well try to recover it by using a live-cd.
> Can you show me the xml file of the domain?
>
> On 11 Nov 2013, at 23:59, Narayanan, Krishnaprasad 
> <narayana at uni-mainz.de> wrote:
>
>> It’s a snapshot and the VM is trying to boot from the snapshot. 
>> It’s a linux image.
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 08:53
>> To: Narayanan, Krishnaprasad
>> Cc: openstack at lists.openstack.org
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> Is it a Linux or a W$ image?
>>
>> On 11 Nov 2013, at 23:29, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>> It returned “No errors were found on the image”.
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 08:22
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> it’s a different problem now.
>> Can you run :
>> qemu-img check /var/lib/nova/[…]/disk
>>
>> thanks !
>>
>> On 11 Nov 2013, at 23:14, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>> In the virsh list, I could see the instance status “running”. The 
>> GUI seems to be synchronized with the data from virsh. I don’t see 
>> an error now. In the VNC I could see an error message “Booting from 
>> floppy. Boot failed:could not read the boot disk. No bootable 
>> device”.
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 08:12
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> if you run:
>> virsh list from the node, is the instance started?
>>
>> On 11 Nov 2013, at 22:56, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>> I am able to start the VM in virsh. I got a new entry in the GUI but 
>> it said Error Spawning.
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 07:31
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> Yes, give the files to the nova.nova user:
>> chown nova: /var/lib/nova/etc…
>>
>> On 11 Nov 2013, at 22:30, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>> Do I have to change the ownership of these two files?
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 07:29
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> yup, should be good :
>> # root > qemu-img create -f qcow2 
>> /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_0_None 
>> 0G
>> On 11 Nov 2013, at 22:28, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>>
>> I created the base image using the command below. I got a new error 
>> “Unable to allow access for disk path 
>> /var/lib/nova/instances/_base/ephemeral_0_20_None: No such file or 
>> directory”.
>>
>> Can I use a similar command to create the ephemeral disk?
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 07:22
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> it’s only an empty file provisioned according to your flavour.
>> run this as root:
>> # root > qemu-img create -f qcow2 
>> /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_20 
>> 20G
>>
>> (pay attention to the last characters, I added the size “20G”)
>>
>> Razique
>>
>>
>> On 11 Nov 2013, at 22:02, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>>
>>
>> Oh I am sorry. I found the disk. You can ignore my previous email.
>>
>> I see a new error now
>> “error: Failed to start domain instance-0000041c
>> error: Unable to allow access for disk path 
>> /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_20: 
>> No such file or directory”
>>
>> This means that the base image is missing. Can I know is it possible 
>> to get this file fc1ff06266a986df09d4293a0be3dbb18884dc1d_20?
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 06:49
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> The _base directory contains only the base (or backing files) images, 
>> from which qemu-kvm creates the instance images.
>> Everytime you spawn an instance, OpenStack creates a copy (not 
>> exactly a copy) of the image. It puts the base image inside _base (if 
>> it doesn’t exist) and create a light image that inherits the 
>> changes (the instance) inside 
>> /var/lib/nova/instance/instance-UUID/disk
>>
>> the disk is linked to that backing file.
>>
>> If you haven’t terminated it though, the file “disk” should be 
>> there. Is it an instance you’ve been able to use or a new one 
>> (meaning you never used it)?
>>
>> On 11 Nov 2013, at 21:45, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>> I only rebooted the instance using the GUI. The instance was never 
>> terminated. Can I know what is present inside the _base folder in 
>> /var/lib/nova/instances?
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 06:40
>> To: Narayanan, Krishnaprasad
>> Cc: openstack
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> That means the disk no longer exists. Did you terminated the 
>> instance?
>> if so, then you won’t be able to retrieve it. Maybe you have 
>> backups?
>>
>> On 11 Nov 2013, at 21:38, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I did as what you had suggested. I see an different error message 
>> now.
>>
>> “error: Failed to start domain instance-UUID
>> error: Unable to allow access for disk path 
>> /var/lib/nova/instances/instance-UUID/disk: No such file or 
>> directory”
>>
>> This means that the disk is missing. Is it possible to create a disk?
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Dienstag, 12. November 2013 05:24
>> To: Narayanan, Krishnaprasad
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> No,
>> if you only need the file, create it manually using the prompt:
>> root #> touch /var/lib/nova/instances/instance-UUID/console.log
>> root # > chown nova. !$
>>
>> On 11 Nov 2013, at 20:02, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> If I get it correct, it has to be created from the virsh console. Can 
>> I know is the following procedure correct for the creation of the 
>> instance
>> a)      dumpxml the instance that is Shutoff
>> b)      create an instance using the output of a)
>>
>> If this is the case, it wouldn’t be recognized in OpenStack. Am I 
>> missing any link here?
>>
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Montag, 11. November 2013 19:51
>> To: Narayanan, Krishnaprasad
>> Cc: openstack at lists.openstack.org
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> Try to create it and chown it to nova:
>>
>>
>>
>> On Monday, November 11, 2013, Narayanan, Krishnaprasad wrote:
>> Hi Razique,
>>
>> I executed the start command on the compute node using the domain 
>> name option. I got an error message saying that
>> Error:“Failed to start domain instance-UUID”
>> Error: Unable to precreate char dev file ‘/var/lib/nova/instances/ 
>> instance-UUID/console.log: No such file or directory.
>>
>> Can I get help to get rid of this problem?
>>
>> Thanks,
>> Krishnaprasad
>> From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
>> Sent: Montag, 11. November 2013 17:22
>> To: Narayanan, Krishnaprasad
>> Cc: openstack at lists.openstack.org
>> Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX
>>
>> You have many solutions for that
>> first try to power it on via
>> $ virsh start
>> and check /var/log/libvritd.log
>>
>> you can use kpartx to discover the partitions within the images, and 
>> use the different builtin system tools to fix the partitions
>> fsck.ext4/ xfs_adm, etc…
>>
>>
>> Razique
>>
>> On 11 Nov 2013, at 04:14, Narayanan, Krishnaprasad 
>> <narayana at uni-mainz.de> wrote:
>>
>>
>> Hallo all,
>>
>> I looked in to forums about recovering VMs in OpenStack ESSEX.  In 
>> the dashboard, I could see that the VMs status is “Hard 
>> rebooting” with power state Shutoff.  In virsh, I could see that 
>> the state is Shutoff. The complete process is taking longer time to 
>> finish and I assume it is in a unrecoverable state. Can anybody help 
>> me in recovering the data on the VM and the VM itself?
>>
>> Thanks,
>> Krishnaprasad
>> _______________________________________________
>> Mailing list: 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131112/f5900f2b/attachment.html>


More information about the Openstack mailing list