Open Stack

On Fri, Aug 14, 2015 at 03:34:09PM +0100, Daniel P. Berrange wrote:
> On Fri, Aug 14, 2015 at 03:10:13PM +0100, Daniel P. Berrange wrote:
> > On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote:
> > > On 08/13/2015 11:37 PM, Ivan Derbenev wrote:
> > > >*From:*Ivan Derbenev [mailto:ivan.derbenev at tech-corps.com]
> > > >*Sent:* Wednesday, August 5, 2015 1:21 PM
> > > >*To:* openstack at lists.openstack.org
> > > >*Subject:* [Openstack] New instances booting time
> > > >
> > > >Hello, guys, I have a question
> > > >
> > > >We now have OS Kilo + KVM+ Ubuntu 14.04
> > > >
> > > >Nova-compute.conf:
> > > >
> > > >[libvirt]
> > > >
> > > >virt_type=kvm
> > > >
> > > >images_type = lvm
> > > >
> > > >images_volume_group =openstack-controller01-ky01-vg
> > > >
> > > >volume_clear = none
> > > >
> > > >the problem is when nova boots a new instance it’s SUPER slow
> > > >
> > > >exactly this step:
> > > >
> > > >Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
> > > >qemu-img convert -O raw
> > > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e
> > > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk
> > > >
> > > >Well, I understand what this step is doing – it’s copying raw image to lvm.
> > > >
> > > >How can we speed it up?
> > > >
> > > >I don’t wanna have instance from 100GB image booted for 4 hours
> > > 
> > > Don't use base images that are 100G in size. Quite simple, really.
> > 
> > Presumably the actual image is only a few 100 MB in size, but the
> > virtual qcow2 size is 100 GB ?
> > 
> > I guess qemu-img probably has to still write out 100 GB of zeros
> > as it can't assume that the LVM target is pre-zeroed. I do wonder
> > though if there's a way to optimize this so that qemu-img only
> > has to write out the 100 MB of actual data, and not al the zeros.
> > 
> > Perhaps there's scope for a flag to qemu-img to tell it to seek
> > in the target when it sees holes in the source image. If OpenStack
> > then used sparse LVM images, there would be no need to rwite out
> > 100's of GB of zeros.
> 
> Speaking with the KVM block layer maintainers, the behaviour seen
> is not entirely expected. qemu-img wants to ensure no existing
> data is left behind as that could cause data corruption if the
> original qcow2 file had dropped empty sectors. So it has to make
> sure it can write out the 100 GB even if that is mostly zeros.
> 
> That said, it has support for using discard, and it will use that
> in preference to writing zeros, which should be really fast. For
> it to work with block devices though, it seems that we need to
> disable the I/O cache by passing '-t none' to qemu-img when doing
> this conversion. This also assumes the underlying storage that
> the user has configured behind their LVM volume supports discard
> which may not always be the case.
> 
> So it would be interesting to know if changing
> 
> $ qemu-img convert -O raw \
>     /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \
>     /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk
> 
> to
> 
> $ qemu-img convert -O raw -t none \
>     /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \
>     /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk
> 
> makes any significant difference to the speed.

I filed a bug to track this possible enhancement

  https://bugs.launchpad.net/nova/+bug/1484992

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|