<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Chris, you are right – and I think there are several synchronization issues in the code there.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The migration process should be synchronized as it is in build_run_instance etc. At the moment if pre_live_migrate fails due to an RPC timeout (i.e. it takes
too long for some reason) the rollback can be initiated while it is still running. So you can get volume attachments and vif plugging being built and torn down at the same time, not good. [I’m not sure if that has a bug report – I’ll file one if not].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We need to pay attention to these cases as we refactor the live migration process. This has begun with:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/async-live-migration-rest-check.html">https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/async-live-migration-rest-check.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/live_migration_compute_communcation.html">https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/live_migration_compute_communcation.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> lương hữu tuấn [mailto:tuantuluong@gmail.com]
<br>
<b>Sent:</b> 10 June 2016 10:20<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> Re: [openstack-dev] [nova] theoretical race between live migration and resource audit?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Yes, it is actually a race and we have already faced a negative effect when using evacuation. Some information of cpu pinning is lost. Imagine that, in some cases, we do some re-scheduling actions (evacuate, live-migration, etc.) then immediately
do the next actions (delete, resize, etc.) before the resource_tracker updates in the next period. In this case, it fails. Actually, it has some negative side in writing tests based on the scenario described above.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Br,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Tutj<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Jun 10, 2016 at 10:36 AM, Matthew Booth <<a href="mailto:mbooth@redhat.com" target="_blank">mbooth@redhat.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Yes, this is a race.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However, it's my understanding that this is 'ok'. The resource tracker doesn't claim to be 100% accurate at all times, right? Otherwise why would it update itself in a period task in the first place. It's my understanding that the resource
tracker is basically a best effort cache, and that scheduling decisions can still fail at the host. The resource tracker will fix itself next time it runs via its periodic task.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Matt (not a scheduler person)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Jun 9, 2016 at 10:41 PM, Chris Friesen <<a href="mailto:chris.friesen@windriver.com" target="_blank">chris.friesen@windriver.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi,<br>
<br>
I'm wondering if we might have a race between live migration and the resource audit. I've included a few people on the receiver list that have worked directly with this code in the past.<br>
<br>
In _update_available_resource() we have code that looks like this:<br>
<br>
instances = objects.InstanceList.get_by_host_and_node()<br>
self._update_usage_from_instances()<br>
migrations = objects.MigrationList.get_in_progress_by_host_and_node()<br>
self._update_usage_from_migrations()<br>
<br>
<br>
In post_live_migration_at_destination() we do this (updating the host and node as well as the task state):<br>
instance.host = self.host<br>
instance.task_state = None<br>
instance.node = node_name<br>
instance.save(expected_task_state=task_states.MIGRATING)<br>
<br>
<br>
And in _post_live_migration() we update the migration status to "completed":<br>
if migrate_data and migrate_data.get('migration'):<br>
migrate_data['migration'].status = 'completed'<br>
migrate_data['migration'].save()<br>
<br>
<br>
Both of the latter routines are not serialized by the COMPUTE_RESOURCE_SEMAPHORE, so they can race relative to the code in _update_available_resource().<br>
<br>
<br>
I'm wondering if we can have a situation like this:<br>
<br>
1) migration in progress<br>
2) We start running _update_available_resource() on destination, and we call instances = objects.InstanceList.get_by_host_and_node(). This will not return the migration, because it is not yet on the destination host.<br>
3) The migration completes and we call post_live_migration_at_destination(), which sets the host/node/task_state on the instance.<br>
4) In _update_available_resource() on destination, we call migrations = objects.MigrationList.get_in_progress_by_host_and_node(). This will return the migration for the instance in question, but when we run self._update_usage_from_migrations() the uuid will
not be in "instances" and so we will use the instance from the newly-queried migration. We will then ignore the instance because it is not in a "migrating" state.<br>
<br>
Am I imagining things, or is there a race here? If so, the negative effects would be that the resources of the migrating instance would be "lost", allowing a newly-scheduled instance to claim the same resources (PCI devices, pinned CPUs, etc.)<br>
<br>
Chris<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span class="hoenzb"><span style="color:#888888">-- <o:p></o:p></span></span></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;color:#888888">Matthew Booth</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Red Hat Engineering, Virtualisation Team<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Phone: <a href="tel:%2B442070094448" target="_blank">
+442070094448</a> (UK)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>