Open Stack

Hi, Ian,
Thanks a lot for your reply.

>In general, while you've applied this to networking (and it's not the first time I've seen this proposal), the same technique will work with any device - PF or VF, networking or other:
>- notify the VM via an accepted channel that a device is going to be temporarily removed
>- remove the device
>- migrate the VM
>- notify the VM that the device is going to be returned
>- reattach the device
>Note that, in the above, I've not used said 'PF', 'VF', 'NIC' or 'qemu'.
Yes, I absolutely agree with you and sorry for my vague expression about that.
Actually, now we just attempt to support the live migration of the instance which directly connected to the passthru VF.
Although the devices what you mentioned all we should implement but that needs a step-by-step plan, I think.


>You would need to document what assumptions the guest is going to make (the reason I mention this is I think it's safe to assume the device has been recently reset here, but for a network device you might want to consider whether the device will have the same MAC address or number of tx and rx buffers, for instance).
Exactly correct, there are a lot things that should be considered, but with regard to VF,
many things will be easy to handle or avoid, for instance, the issuse of same MAC address.

And in addition to what you mentioned, for VF, I think, the most important thing we need to discuss
is the strategy of NIC bonding - How and When and by whom to make this bonding - as there are
too much risks of running afoul of something on the VM, for instance NetworkManager, which will cause failure of bonding presumably.
For instance:
  - prospectively make the NICs bonding when the VM was launched by an embedded script based on DIB?
  - or manually bond the NICs by VM administrators before the live-migration command executes?
  - or notify the VM to perform this bonding by openstack components while the live-migration command executes?
-  ...


Best regards,
Xiexs


From: Ian Wells [mailto:ijw.ubuntu at cack.org.uk]
Sent: Wednesday, February 17, 2016 3:24 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][neutron] New BP for live migration with direct pci passthru

In general, while you've applied this to networking (and it's not the first time I've seen this proposal), the same technique will work with any device - PF or VF, networking or other:
- notify the VM via an accepted channel that a device is going to be temporarily removed
- remove the device
- migrate the VM
- notify the VM that the device is going to be returned
- reattach the device
Note that, in the above, I've not used said 'PF', 'VF', 'NIC' or 'qemu'.

You would need to document what assumptions the guest is going to make (the reason I mention this is I think it's safe to assume the device has been recently reset here, but for a network device you might want to consider whether the device will have the same MAC address or number of tx and rx buffers, for instance).

The method of notification I've deliberately skipped here; you have an answer for qemu, qemu is not the only hypervisor in the world so this will clearly be variable.  A metadata server mechanism is another possibility.

Half of what you've described is one model of how the VM might choose to deal with that (and a suggestion that's come up before, in fact) - that's a model we would absolutely want Openstack to support (and I think the above is sufficient to support it), but we can't easily mandate how VMs behave, so from the Openstack perspective it's more a recommendation than anything we can code up.


On 15 February 2016 at 23:25, Xie, Xianshan <xiexs at cn.fujitsu.com<mailto:xiexs at cn.fujitsu.com>> wrote:
Hi, Fawad,


> Can you please share the link?
https://blueprints.launchpad.net/nova/+spec/direct-pci-passthrough-live-migration

Thanks in advance.


Best regards,
xiexs

From: Fawad Khaliq [mailto:fawad at plumgrid.com<mailto:fawad at plumgrid.com>]
Sent: Tuesday, February 16, 2016 1:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][neutron] New BP for live migration with direct pci passthru

On Mon, Feb 1, 2016 at 3:25 PM, Xie, Xianshan <xiexs at cn.fujitsu.com<mailto:xiexs at cn.fujitsu.com>> wrote:
Hi, all,
  I have registered a new BP about the live migration with a direct pci passthru device.
  Could you please help me to review it? Thanks in advance.

Can you please share the link?


The following is the details:
----------------------------------------------------------------------------------
SR-IOV has been supported for a long while, in the community's point of view,
the pci passthru with Macvtap can be live migrated possibly, but the direct pci passthru
seems hard to implement the migration as the passthru VF is totally controlled by
the VMs so that some internal states may be unknown by the hypervisor.

But we think the direct pci passthru model can also be live migrated with the
following combination of a series of technology/operation based on the enhanced
Qemu-Geust-Agent(QGA) which has already been supported by nova.
   1)Bond the direct pci passthru NIC with a virtual NIC.
     This will keep the network connectivity during the live migration.
   2)Unenslave the direct pci passthru NIC
   3)Hot-unplug the direct pci passthru NIC
   4)Live-migrate guest with the virtual NIC
   5)Hot-plug the direct pci passthru NIC on the target host
   6)Enslave the direct pci passthru NIC

And more inforation about this concept can refer to [1].
[1]https://www.kernel.org/doc/ols/2008/ols2008v2-pages-261-267.pdf
----------------------------------------------------------------------------------

Best regards,
Xiexs



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160219/0b5865af/attachment-0001.html>