Open Stack

On Fri, Aug 14, 2015 at 03:10:13PM +0100, Daniel P. Berrange wrote:
> On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote:
> > On 08/13/2015 11:37 PM, Ivan Derbenev wrote:
> > >*From:*Ivan Derbenev [mailto:ivan.derbenev at tech-corps.com]
> > >*Sent:* Wednesday, August 5, 2015 1:21 PM
> > >*To:* openstack at lists.openstack.org
> > >*Subject:* [Openstack] New instances booting time
> > >
> > >Hello, guys, I have a question
> > >
> > >We now have OS Kilo + KVM+ Ubuntu 14.04
> > >
> > >Nova-compute.conf:
> > >
> > >[libvirt]
> > >
> > >virt_type=kvm
> > >
> > >images_type = lvm
> > >
> > >images_volume_group =openstack-controller01-ky01-vg
> > >
> > >volume_clear = none
> > >
> > >the problem is when nova boots a new instance it’s SUPER slow
> > >
> > >exactly this step:
> > >
> > >Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
> > >qemu-img convert -O raw
> > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e
> > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk
> > >
> > >Well, I understand what this step is doing – it’s copying raw image to lvm.
> > >
> > >How can we speed it up?
> > >
> > >I don’t wanna have instance from 100GB image booted for 4 hours
> > 
> > Don't use base images that are 100G in size. Quite simple, really.
> 
> Presumably the actual image is only a few 100 MB in size, but the
> virtual qcow2 size is 100 GB ?
> 
> I guess qemu-img probably has to still write out 100 GB of zeros
> as it can't assume that the LVM target is pre-zeroed. I do wonder
> though if there's a way to optimize this so that qemu-img only
> has to write out the 100 MB of actual data, and not al the zeros.
> 
> Perhaps there's scope for a flag to qemu-img to tell it to seek
> in the target when it sees holes in the source image. If OpenStack
> then used sparse LVM images, there would be no need to rwite out
> 100's of GB of zeros.

Speaking with the KVM block layer maintainers, the behaviour seen
is not entirely expected. qemu-img wants to ensure no existing
data is left behind as that could cause data corruption if the
original qcow2 file had dropped empty sectors. So it has to make
sure it can write out the 100 GB even if that is mostly zeros.

That said, it has support for using discard, and it will use that
in preference to writing zeros, which should be really fast. For
it to work with block devices though, it seems that we need to
disable the I/O cache by passing '-t none' to qemu-img when doing
this conversion. This also assumes the underlying storage that
the user has configured behind their LVM volume supports discard
which may not always be the case.

So it would be interesting to know if changing

$ qemu-img convert -O raw \
    /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \
    /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk

to

$ qemu-img convert -O raw -t none \
    /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \
    /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk

makes any significant difference to the speed.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|