<div dir="ltr">I think we can add a config option for this and set a theoretical proper default value, <div>we also add help messages to inform the the user about how inappropriate value of</div><div>this config option will effect the performance.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 3, 2016 at 7:45 PM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Wed, Feb 03, 2016 at 11:27:16AM +0000, Paul Carlton wrote:<br>
> On 03/02/16 10:49, Daniel P. Berrange wrote:<br>
> >On Wed, Feb 03, 2016 at 10:44:36AM +0000, Daniel P. Berrange wrote:<br>
> >>On Wed, Feb 03, 2016 at 10:37:24AM +0000, Koniszewski, Pawel wrote:<br>
> >>>Hello everyone,<br>
> >>><br>
> >>>On the yesterday's live migration meeting we had concerns that interval of<br>
> >>>writing migration progress to the database is too short.<br>
> >>><br>
> >>>Information about migration progress will be stored in the database and<br>
> >>>exposed through the API (/servers/<uuid>/migrations/<id>). In current<br>
> >>>proposition [1] migration progress will be updated every 2 seconds. It<br>
> >>>basically means that every 2 seconds a call through RPC will go from compute<br>
> >>>to conductor to write migration data to the database. In case of parallel<br>
> >>>live migrations each migration will report progress by itself.<br>
> >>><br>
> >>>Isn't 2 seconds interval too short for updates if the information is exposed<br>
> >>>through the API and it requires RPC and DB call to actually save it in the<br>
> >>>DB?<br>
> >>><br>
> >>>Our default configuration allows only for 1 concurrent live migration [2],<br>
> >>>but it might vary between different deployments and use cases as it is<br>
> >>>configurable. Someone might want to trigger 10 (or even more) parallel live<br>
> >>>migrations and each might take even a day to finish in case of block<br>
> >>>migration. Also if deployment is big enough rabbitmq might be fully-loaded.<br>
> >>>I'm not sure whether updating each migration every 2 seconds makes sense in<br>
> >>>this case. On the other hand it might be hard to observe fast enough that<br>
> >>>migration is stuck if we increase this interval...<br>
> >>Do we have any actual data that this is a real problem. I have a pretty hard<br>
> >>time believing that a database update of a single field every 2 seconds is<br>
> >>going to be what pushes Nova over the edge into a performance collapse, even<br>
> >>if there are 20 migrations running in parallel, when you compare it to the<br>
> >>amount of DB queries & updates done across other areas of the code for pretty<br>
> >>much every singke API call and background job.<br>
> >Also note that progress is rounded to the nearest integer. So even if the<br>
> >migration runs all day, there is a maximum of 100 possible changes in value<br>
> >for the progress field, so most of the updates should turn in to no-ops at<br>
> >the database level.<br>
> ><br>
> >Regards,<br>
> >Daniel<br>
> I agree with Daniel, these rpc and db access ops are a tiny percentage<br>
> of the overall load on rabbit and mysql and properly configured these<br>
> subsystems should have no issues with this workload.<br>
><br>
> One correction, unless I'm misreading it, the existing<br>
> _live_migration_monitor code updates the progress field of the instance<br>
> record every 5 seconds. However this value can go up and down so<br>
> an infinate number of updates are possible?<br>
<br>
</div></div>Oh yes, you are in fact correct. Technically you could have an unbounded<br>
number of updates if migration goes backwards. Some mitigation against<br>
this is if we see progress going backwards we'll actually abort the<br>
migration if it gets stuck for too long. We'll also be progressively<br>
increasing the permitted downtime. So except in pathelogical scenarios<br>
I think the number of updates should still be relatively small.<br>
<span class=""><br>
> However, the issue raised here is not with the existing implementation<br>
> but with the proposed change<br>
> <a href="https://review.openstack.org/#/c/258813/5/nova/virt/libvirt/driver.py" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/258813/5/nova/virt/libvirt/driver.py</a><br>
> This add a save() operation on the migration object every 2 seconds<br>
<br>
</span>Ok, that is more heavy weight since it is recording the raw byte values<br>
and so it is guaranteed to do a database update pretty much every time.<br>
It still shouldn't be too unreasonable a loading though. FWIW I think<br>
it is worth being consistent in the update frequency betweeen the<br>
progress value & the migration object save, so switching to be every<br>
5 seconds probably makes more sense, so we know both objects are<br>
reflecting the same point in time.<br>
<span class="im HOEnZb"><br>
Regards,<br>
Daniel<br>
--<br>
|: <a href="http://berrange.com" rel="noreferrer" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" rel="noreferrer" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" rel="noreferrer" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" rel="noreferrer" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" rel="noreferrer" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" rel="noreferrer" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" rel="noreferrer" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" rel="noreferrer" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
<br>
</span><div class="HOEnZb"><div class="h5">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>