<div dir="ltr"><div><div><div><div><div>In general, while you've applied this to
networking (and it's not the first time I've seen this proposal), the
same technique will work with any device - PF or VF, networking or
other:<br><br></div>- notify the VM via an accepted channel that a
device is going to be temporarily removed<br></div>- remove the device<br></div>- migrate the VM<br></div><div>- notify the VM that the device is going to be returned<br></div>- reattach the device<br><br></div><div>Note that, in the above, I've not used said 'PF', 'VF', 'NIC' or 'qemu'.<br></div><div><br></div><div>You
would need to document what assumptions the guest is going to make (the
reason I mention this is I think it's safe to assume the device has
been recently reset here, but for a network device you might want to
consider whether the device will have the same MAC address or number of
tx and rx buffers, for instance).<br></div><div><br></div><div>The
method of notification I've deliberately skipped here; you have an
answer for qemu, qemu is not the only hypervisor in the world so this
will clearly be variable. A metadata server mechanism is another
possibility.<br></div><div><br></div>Half of what you've described is
one model of how the VM might choose to deal with that (and a suggestion that's come up before, in fact) - that's a model we
would absolutely want Openstack to support (and I think the above is sufficient to support it), but we can't easily mandate
how VMs behave, so from the Openstack perspective it's more a
recommendation than anything we can code up.<br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 15 February 2016 at 23:25, Xie, Xianshan <span dir="ltr"><<a href="mailto:xiexs@cn.fujitsu.com" target="_blank">xiexs@cn.fujitsu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="ZH-CN">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Hi, Fawad,<u></u><u></u></span></p><span class="">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">></span><span lang="EN-US"> Can you please share the link?</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u><u></u></span></p>
</span><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><a href="https://blueprints.launchpad.net/nova/+spec/direct-pci-passthrough-live-migration" target="_blank">https://blueprints.launchpad.net/nova/+spec/direct-pci-passthrough-live-migration</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Thanks in advance.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">xiexs<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Fawad Khaliq [mailto:<a href="mailto:fawad@plumgrid.com" target="_blank">fawad@plumgrid.com</a>]
<br>
<b>Sent:</b> Tuesday, February 16, 2016 1:19 PM<span class=""><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [nova][neutron] New BP for live migration with direct pci passthru<u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Feb 1, 2016 at 3:25 PM, Xie, Xianshan <<a href="mailto:xiexs@cn.fujitsu.com" target="_blank">xiexs@cn.fujitsu.com</a>> wrote:<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><span lang="EN-US">Hi, all,<br>
I have registered a new BP about the live migration with a direct pci passthru device.<br>
Could you please help me to review it? Thanks in advance.<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Can you please share the link? <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>
The following is the details:<br>
----------------------------------------------------------------------------------<br>
SR-IOV has been supported for a long while, in the community's point of view,<br>
the pci passthru with Macvtap can be live migrated possibly, but the direct pci passthru<br>
seems hard to implement the migration as the passthru VF is totally controlled by<br>
the VMs so that some internal states may be unknown by the hypervisor.<br>
<br>
But we think the direct pci passthru model can also be live migrated with the<br>
following combination of a series of technology/operation based on the enhanced<br>
Qemu-Geust-Agent(QGA) which has already been supported by nova.<br>
1)Bond the direct pci passthru NIC with a virtual NIC.<br>
This will keep the network connectivity during the live migration.<br>
2)Unenslave the direct pci passthru NIC<br>
3)Hot-unplug the direct pci passthru NIC<br>
4)Live-migrate guest with the virtual NIC<br>
5)Hot-plug the direct pci passthru NIC on the target host<br>
6)Enslave the direct pci passthru NIC<br>
<br>
And more inforation about this concept can refer to [1].<br>
[1]<a href="https://www.kernel.org/doc/ols/2008/ols2008v2-pages-261-267.pdf" target="_blank">https://www.kernel.org/doc/ols/2008/ols2008v2-pages-261-267.pdf</a><br>
----------------------------------------------------------------------------------<br>
<br>
Best regards,<br>
Xiexs<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></span></p>
</blockquote>
</div></div></div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
</div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>