Open Stack

Thanks for bringing this up, Daniel.  I don't think it makes sense to have
a timeout on live migration, but operators should be able to cancel it,
just like any other unbounded long-running process.  For example, there's
no timeout on file transfers, but they need an interface report progress
and to cancel them.  That would imply an option to cancel evacuation too.

--
Noel


On Fri, Jan 30, 2015 at 8:47 AM, Daniel P. Berrange <berrange at redhat.com>
wrote:

> In working on a recent Nova migration bug
>
>   https://bugs.launchpad.net/nova/+bug/1414065
>
> I had cause to refactor the way the nova libvirt driver monitors live
> migration completion/failure/progress. This refactor has opened the
> door for doing more intelligent active management of the live migration
> process.
>
> As it stands today, we launch live migration, with a possible bandwidth
> limit applied and just pray that it succeeds eventually. It might take
> until the end of the universe and we'll happily wait that long. This is
> pretty dumb really and I think we really ought to do better. The problem
> is that I'm not really sure what "better" should mean, except for ensuring
> it doesn't run forever.
>
> As a demo, I pushed a quick proof of concept showing how we could easily
> just abort live migration after say 10 minutes
>
>   https://review.openstack.org/#/c/151665/
>
> There are a number of possible things to consider though...
>
> First how to detect when live migration isn't going to succeeed.
>
>  - Could do a crude timeout, eg allow 10 minutes to succeeed or else.
>
>  - Look at data transfer stats (memory transferred, memory remaining to
>    transfer, disk transferred, disk remaining to transfer) to determine
>    if it is making forward progress.
>
>  - Leave it upto the admin / user to decided if it has gone long enough
>
> The first is easy, while the second is harder but probably more reliable
> and useful for users.
>
> Second is a question of what todo when it looks to be failing
>
>  - Cancel the migration - leave it running on source. Not good if the
>    admin is trying to evacuate a host.
>
>  - Pause the VM - make it complete as non-live migration. Not good if
>    the guest workload doesn't like being paused
>
>  - Increase the bandwidth permitted. There is a built-in rate limit in
>    QEMU overridable via nova.conf. Could argue that the admin should just
>    set their desired limit in nova.conf and be done with it, but perhaps
>    there's a case for increasing it in special circumstances. eg emergency
>    evacuate of host it is better to waste bandwidth & complete the job,
>    but for non-urgent scenarios better to limit bandwidth & accept failure
> ?
>
>  - Increase the maximum downtime permitted. This is the small time window
>    when the guest switches from source to dest. To small and it'll never
>    switch, too large and it'll suffer unacceptable interuption.
>
> We could do some of these things automatically based on some policy
> or leave them upto the cloud admin/tenant user via new APIs
>
> Third there's question of other QEMU features we could make use of to
> stop problems in the first place
>
>  - Auto-converge flag - if you set this QEMU throttles back the CPUs
>    so the guest cannot dirty ram pages as quickly. This is nicer than
>    pausing CPUs altogether, but could still be an issue for guests
>    which have strong performance requirements
>
>  - Page compression flag - if you set this QEMU does compression of
>    pages to reduce data that has to be sent. This is basically trading
>    off network bandwidth vs CPU burn. Probably a win unless you are
>    already highly overcomit on CPU on the host
>
> Fourth there's a question of whether we should give the tenant user or
> cloud admin further APIs for influencing migration
>
>  - Add an explicit API for cancelling migration ?
>
>  - Add APIs for setting tunables like downtime, bandwidth on the fly ?
>
>  - Or drive some of the tunables like downtime, bandwidth, or policies
>    like cancel vs paused from flavour or image metadata properties ?
>
>  - Allow operations like evacuate to specify a live migration policy
>    eg switch non-live migrate after 5 minutes ?
>
> The current code is so crude and there's a hell of alot of options we
> can take. I'm just not sure which is the best direction for us to go
> in.
>
> What kind of things would be the biggest win from Operators' or tenants'
> POV ?
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org              -o-             http://virt-manager.org
> :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc
> :|
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150201/2cd71c83/attachment.html>