Open Stack

> -----Original Message-----
> From: Daniel P. Berrange [mailto:berrange at redhat.com]
> Sent: 03 February 2016 10:49
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: Feng, Shaohe
> Subject: Re: [openstack-dev] [nova] Migration progress
> 
> On Wed, Feb 03, 2016 at 10:44:36AM +0000, Daniel P. Berrange wrote:
> > On Wed, Feb 03, 2016 at 10:37:24AM +0000, Koniszewski, Pawel wrote:
> > > Hello everyone,
> > >
> > > On the yesterday's live migration meeting we had concerns that
> > > interval of writing migration progress to the database is too short.
> > >
> > > Information about migration progress will be stored in the database
> > > and exposed through the API (/servers/<uuid>/migrations/<id>). In
> > > current proposition [1] migration progress will be updated every 2
> > > seconds. It basically means that every 2 seconds a call through RPC
> > > will go from compute to conductor to write migration data to the
> > > database. In case of parallel live migrations each migration will report
> progress by itself.
> > >
> > > Isn't 2 seconds interval too short for updates if the information is
> > > exposed through the API and it requires RPC and DB call to actually
> > > save it in the DB?
> > >
> > > Our default configuration allows only for 1 concurrent live
> > > migration [2], but it might vary between different deployments and
> > > use cases as it is configurable. Someone might want to trigger 10
> > > (or even more) parallel live migrations and each might take even a
> > > day to finish in case of block migration. Also if deployment is big enough
> rabbitmq might be fully-loaded.
> > > I'm not sure whether updating each migration every 2 seconds makes
> > > sense in this case. On the other hand it might be hard to observe
> > > fast enough that migration is stuck if we increase this interval...
> >
> > Do we have any actual data that this is a real problem. I have a
> > pretty hard time believing that a database update of a single field
> > every 2 seconds is going to be what pushes Nova over the edge into a
> > performance collapse, even if there are 20 migrations running in
> > parallel, when you compare it to the amount of DB queries & updates
> > done across other areas of the code for pretty much every singke API call
> and background job.

As a data point: when we were doing live migrations in HP public cloud for rolling updates we were maintaining approximately 150 concurrent migrations through the process. At 2s intervals that would make approx. 75 updates per second. We don't feel that would have been a problem.

We also spoke to Michael Still and he thought it wouldn't be a problem for Rack Space (remembering they have cells). Having said that I have no idea of numbers I their case and would rather they spoke for themselves. In this thread.



> 
> Also note that progress is rounded to the nearest integer. So even if the
> migration runs all day, there is a maximum of 100 possible changes in value
> for the progress field, so most of the updates should turn in to no-ops at the
> database level.
> 
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
> 
> __________________________________________________________
> ________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev