Open Stack

The guestfs error is due to a misconfiguration of guestfs and will only result in not having files injected before booting. Usually this is harmless as cloud-init like software will fix it after booting.

Glad the VM booted ☺


From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 05 March 2014 15:10
To: Parthipan, Loganathan
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

changing /etc/libvirt/qemu.conf helped [Image removed by sender. Emoji] . clear_emulator_capabilities=0 was already set.

VM boots up, but it still has the same error in log.

log <http://paste.openstack.org/raw/72621/>

instance-log<http://paste.openstack.org/raw/72622/>

ls -alh /var/lib/nova/instances/7d79e021-72a9-4e5c-b2e5-312e16355dcd/
total 52M
drwxr-xr-x 2 nova nova 4.0K Mar  5 15:50 .
drwxr-xr-x 5 nova nova 4.0K Mar  5 15:50 ..
-rw-rw---- 1 root root  25K Mar  5 15:51 console.log
-rw-r--r-- 1 root root  52M Mar  5 15:51 disk
-rw-r--r-- 1 nova nova 1.6K Mar  5 15:50 libvirt.xml

________________________________
From: parthipan at hp.com
To: arindam at live.com; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Wed, 5 Mar 2014 13:53:25 +0000
If that doesn’t work try setting clear_emulator_capabilities=0

From: Parthipan, Loganathan
Sent: 05 March 2014 13:37
To: 'Arindam Choudhury'; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Try setting user and group to root in /etc/libvirt/qemu.conf

From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 05 March 2014 13:36
To: Parthipan, Loganathan; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)


./configure --enable-trace-backend=ftrace --enable-kvm --enable-vhost-net --enable-virtfs --target-list=x86_64-linux-user,x86_64-softmmu
________________________________
From: parthipan at hp.com
To: arindam at live.com; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Wed, 5 Mar 2014 13:30:18 +0000
What were your build time options for qemu?

From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 05 March 2014 13:29
To: Parthipan, Loganathan; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

the permission of /dev/kvm:

crw-rw-rw-+ 1 root kvm 10, 232 Mar  4 14:28 /dev/kvm
________________________________
From: parthipan at hp.com
To: arindam at live.com; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Wed, 5 Mar 2014 13:23:41 +0000
What are /dev/kvm permissions?


From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 05 March 2014 11:09
To: Parthipan, Loganathan; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Which user I must add in root group? nova or qemu?
________________________________
From: parthipan at hp.com
To: arindam at live.com; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Wed, 5 Mar 2014 10:12:14 +0000
Seems like qemu doesn’t have the privileges to read /sys/kernel/debug/tracing/tracing_on



From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 04 March 2014 15:15
To: Parthipan, Loganathan; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Hi,
both
/sys/kernel/debug/tracing/tracing_on
/sys/kernel/debug/tracing/events/kvm/enable is 1.

The logs are:

libvirtd-debug<http://paste.openstack.org/raw/72197/> (sorry for lots of garbage in this file)

/var/log/messages<http://paste.openstack.org/raw/72198/>

instance.log<http://paste.openstack.org/raw/72205/>

After disabling tracing_on and tracing kvm events, it gave the same error.

Regards,
Arindam
________________________________
From: parthipan at hp.com
To: arindam at live.com
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Tue, 4 Mar 2014 14:07:31 +0000
If you can that’ll be great. Thanks.

Also what’s in

/sys/kernel/debug/tracing/tracing_on
/sys/kernel/debug/tracing/events/kvm/enable

Try running after
echo 0 > /sys/kernel/debug/tracing/tracing_on

If there’s no improvement try - although I don’t know what it really does (I need to read the code)
echo 0 > /sys/kernel/debug/tracing/events/kvm/enable

I’m just wondering if it’s due to some permission/ACL issues when trying to write into debugfs using ftrace, given your observation on different tracing backends.

Thanks for reporting and spending time on this.

Cheers
~parthi


From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 04 March 2014 13:27
To: Parthipan, Loganathan
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Hi,

Its a multi-node deployment with nova-network.

If you want I can recreate the error and send you the logs.

Regards,
________________________________
From: parthipan at hp.com
To: arindam at live.com
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Tue, 4 Mar 2014 13:21:24 +0000
Interesting…

Is it a single node deployment? Do you have any other log data such as libvirt_debug.log, libvirt/qemu/instance-xxxx.log? Any errors/warnings during the time of instance creation in syslog?

From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 04 March 2014 12:56
To: Parthipan, Loganathan
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)


it only happens when I enable

--enable-trace-backend=ftrace

with
--enable-trace-backend=simple

works fine.

________________________________
From: parthipan at hp.com
To: arindam at live.com
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Tue, 4 Mar 2014 12:33:55 +0000
Also check if it’s a cgroups issue since the binary location changed. You could look at the syslog and kern.log


From: Parthipan, Loganathan
Sent: 04 March 2014 12:15
To: 'Arindam Choudhury'; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Reading your log, I don’t think the problem is due to injection failure at libguestfs. It’s a warning and is ignored.

Seems more like a network setup issue. Are you using neutron? Is the network/subnet etc setup properly?

From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 04 March 2014 11:50
To: Parthipan, Loganathan; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Hi Parthipan,

When I try to do yum remove python-libguestfs.
It also wants to remove openstack-nova-compute. So, I can not.

Regards,
Arindam
________________________________
From: parthipan at hp.com
To: arindam at live.com; openstack at lists.openstack.org
Subject: RE: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)
Date: Tue, 4 Mar 2014 11:23:50 +0000
Arindam,

What happens if you ‘apt-get remove python-guestfs’ and try? I’d expect it to fallback to nbdmount.


From: Arindam Choudhury [mailto:arindam at live.com]
Sent: 04 March 2014 10:40
To: openstack at lists.openstack.org
Subject: [Openstack] Error mounting disk with libguestfs (child process died unexpectedly)

Hi,
I have a working Openstack Grizzly setup with nova-network on Scientific Linux. As, SL comes with a old qemu:

qemu-kvm-0.12.1.2-2.355.el6_4.6.x86_64

I am trying to upgrade it to qemu-1.7.0 from source.

The configuration output<http://pastebin.com/raw.php?i=geEEXm79>.

As, openstack use /usr/libexec/qemu-kvm to start up VM, i linked
qemu-kvm to qemu-system-x86_64:
ls -l /usr/libexec/qemu-kvm lrwxrwxrwx 1 root root 33 Feb 19 15:25 /usr/libexec/qemu-kvm -> /usr/local/bin/qemu-system-x86_64

But when I try to start VM, I get following error:

WARNING nova.virt.disk.api [req-7dfd6cf1-db21-403d-abbf-ccb5f0273a6a
b829b65d0675416198f9e4838b5009e2 9a2e060772204260b53b52ea1c12db97]
Ignoring error injecting data into image
(Error mounting /var/lib/nova/instances/07c0898e-7d75-448d-b4fe-2c37f681b846/disk
with libguestfs (child process died unexpectedly))

The full error log is here<http://pastebin.com/raw.php?i=FD8N47mk>.

Output of libguestfs-test-tool is here<http://pastebin.com/raw.php?i=EKWQWH1y>.

Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140305/97e3529e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD000.jpg
Type: image/jpeg
Size: 823 bytes
Desc: ~WRD000.jpg
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140305/97e3529e/attachment.jpg>