Open Stack

Hi Razique,

Thanks for providing me the description and steps. It was helpful. I later discovered from our network administrator that few of our compute node have their instance disk images mounted to a NFS server. I went through the NFS server looking for the below UUIDs, found and fixed the below problem.

I have another question. In case of VM with tiny flavor where the root and ephemeral sizes are 0 GB, can I know what happens during boot failure or how do you recover them?

Thanks,
Krishnaprasad
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 09:41
To: Narayanan, Krishnaprasad
Cc: openstack at lists.openstack.org
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

Ok,
in that case, use kpartx from the host against the disk image to discover partitions:
kpartx -lv /var/lib/nova/[...]/disk
to see if you find any partition

You can as well try to recover it by using a live-cd.
Can you show me the xml file of the domain?

On 11 Nov 2013, at 23:59, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:


It's a snapshot and the VM is trying to boot from the snapshot. It's a linux image.
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 08:53
To: Narayanan, Krishnaprasad
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

Is it a Linux or a W$ image?

On 11 Nov 2013, at 23:29, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:



It returned "No errors were found on the image".

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 08:22
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

it's a different problem now.
Can you run :
qemu-img check /var/lib/nova/[...]/disk

thanks !

On 11 Nov 2013, at 23:14, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:




In the virsh list, I could see the instance status "running". The GUI seems to be synchronized with the data from virsh. I don't see an error now. In the VNC I could see an error message "Booting from floppy. Boot failed:could not read the boot disk. No bootable device".
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 08:12
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

if you run:
virsh list from the node, is the instance started?

On 11 Nov 2013, at 22:56, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:





I am able to start the VM in virsh. I got a new entry in the GUI but it said Error Spawning.
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 07:31
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

Yes, give the files to the nova.nova user:
chown nova: /var/lib/nova/etc...

On 11 Nov 2013, at 22:30, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:






Do I have to change the ownership of these two files?

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 07:29
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

yup, should be good :
# root > qemu-img create -f qcow2 /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_0_None 0G
On 11 Nov 2013, at 22:28, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:







I created the base image using the command below. I got a new error "Unable to allow access for disk path /var/lib/nova/instances/_base/ephemeral_0_20_None: No such file or directory".

Can I use a similar command to create the ephemeral disk?

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 07:22
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

it's only an empty file provisioned according to your flavour.
run this as root:
# root > qemu-img create -f qcow2 /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_20 20G

(pay attention to the last characters, I added the size "20G")

Razique


On 11 Nov 2013, at 22:02, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:








Oh I am sorry. I found the disk. You can ignore my previous email.

I see a new error now
"error: Failed to start domain instance-0000041c
error: Unable to allow access for disk path /var/lib/nova/instances/_base/fc1ff06266a986df09d4293a0be3dbb18884dc1d_20: No such file or directory"

This means that the base image is missing. Can I know is it possible to get this file fc1ff06266a986df09d4293a0be3dbb18884dc1d_20?
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 06:49
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

The _base directory contains only the base (or backing files) images, from which qemu-kvm creates the instance images.
Everytime you spawn an instance, OpenStack creates a copy (not exactly a copy) of the image. It puts the base image inside _base (if it doesn't exist) and create a light image that inherits the changes (the instance) inside /var/lib/nova/instance/instance-UUID/disk

the disk is linked to that backing file.

If you haven't terminated it though, the file "disk" should be there. Is it an instance you've been able to use or a new one (meaning you never used it)?

 On 11 Nov 2013, at 21:45, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:









I only rebooted the instance using the GUI. The instance was never terminated. Can I know what is present inside the _base folder in /var/lib/nova/instances?

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 06:40
To: Narayanan, Krishnaprasad
Cc: openstack
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

That means the disk no longer exists. Did you terminated the instance?
if so, then you won't be able to retrieve it. Maybe you have backups?

On 11 Nov 2013, at 21:38, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:










I did as what you had suggested. I see an different error message now.

"error: Failed to start domain instance-UUID
error: Unable to allow access for disk path /var/lib/nova/instances/instance-UUID/disk: No such file or directory"

This means that the disk is missing. Is it possible to create a disk?

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Dienstag, 12. November 2013 05:24
To: Narayanan, Krishnaprasad
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

No,
if you only need the file, create it manually using the prompt:
root #> touch /var/lib/nova/instances/instance-UUID/console.log
root # > chown nova. !$

On 11 Nov 2013, at 20:02, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:











If I get it correct, it has to be created from the virsh console. Can I know is the following procedure correct for the creation of the instance
a)      dumpxml the instance that is Shutoff
b)      create an instance using the output of a)

If this is the case, it wouldn't be recognized in OpenStack. Am I missing any link here?

From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Montag, 11. November 2013 19:51
To: Narayanan, Krishnaprasad
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

Try to create it and chown it to nova:



On Monday, November 11, 2013, Narayanan, Krishnaprasad wrote:
Hi Razique,

I executed the start command on the compute node using the domain name option. I got an error message saying that
Error:"Failed to start domain instance-UUID"
Error: Unable to precreate char dev file '/var/lib/nova/instances/ instance-UUID/console.log: No such file or directory.

Can I get help to get rid of this problem?

Thanks,
Krishnaprasad
From: Razique Mahroua [mailto:razique.mahroua at gmail.com]
Sent: Montag, 11. November 2013 17:22
To: Narayanan, Krishnaprasad
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] Recovering VMs in OPenStack ESSEX

You have many solutions for that
first try to power it on via
$ virsh start
and check /var/log/libvritd.log

you can use kpartx to discover the partitions within the images, and use the different builtin system tools to fix the partitions
fsck.ext4/ xfs_adm, etc...


Razique

On 11 Nov 2013, at 04:14, Narayanan, Krishnaprasad <narayana at uni-mainz.de<mailto:narayana at uni-mainz.de>> wrote:

Hallo all,

I looked in to forums about recovering VMs in OpenStack ESSEX.  In the dashboard, I could see that the VMs status is "Hard rebooting" with power state Shutoff.  In virsh, I could see that the state is Shutoff. The complete process is taking longer time to finish and I assume it is in a unrecoverable state. Can anybody help me in recovering the data on the VM and the VM itself?

Thanks,
Krishnaprasad
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131112/3872ecf5/attachment.html>