Open Stack

How should we expose Virtual GPUs to Nova?

Various discussions have happened on the original spec submission for Mitaka[1] and the recent submission for Newton[2], however there are a few questions which need further discussion.  But before those question (at the end), some thinking behind the current plans.


*         What sort of model should be use for Virtual GPU provisioning?
Virtual GPUs could be considered to be devices like PCI devices or they could be considered to be a collection of resources.  Some hypervisor implementations (e.g. XenServer) present pre-defined virtual GPU model types to administrators, but Hyper-V's remote FX spec[3] is using a resource-based approach for graphics provisioning.
There is also a lack of consistency between these two approaches in hardware: Some (Intel's GVT-g) could theoretically support per-GPU configuration, but this is not supported in other cases (e.g. NVIDIA's vGPU) and not possible in the case of AMD's MxGPU (which is essentially SR-IOV)

As we want to cope with vendors in the same way, the suggestion is that the hypervisor/driver should expect to expose discrete pre-defined virtual GPU model types, rather than using a collection of resources.


*         Exposing GPU model types to Nova
Assuming that we have pre-defined model types, how do these get exposed to Nova?  The spec proposed for Mitaka spec suggested a new VGPU resource type, however qemu's expected implementation of virtualised GPUs ('mediated devices') is to use VFIO to present PCI-like devices to qemu[4].  In addition, a GPU device could conceptually be passed through to a single VM or split up and passed through to multiple VMs, which is very similar to SR-IOV.  As such, the spec proposed Newton suggested re-using the PCI passthrough code.

The issue comes because these devices are not strictly SR-IOV devices, and may not exist as PCI devices.  Therefore exposing these using the PCI address format introduces complications which either have to be worked around with some 'creative' use of the PCI address format to create the fake PCI devices, or with refactoring the PCI passthrough code to support multiple addressing types, effectively making the PCI passthrough code work with any device available on the host.

We could then have different 'address_type's to represent different types of resources that were to be made available to VMs.

So, some questions:

How much value is there in genericising the PCI passthrough code to support multiple address types?  What other address types might we want to use - e.g. USB?, SCSI?, IDE?

How would we represent an address for virtual GPUs?  Would we define a format to be used by each driver, or allow the driver to specify their own address format?  See also virt-device-role-tagging [5] which uses various addresses for different busses.

The addition of an address_type would likely need a refactor of PCIDevice to be a HostDevice to minimize confusion - How far should that refactor go?  (e.g. should it include renaming the pci_stats column to host_device _stats?)

I'm sure there are other questions... but I can't think of them now! :)

Bob

[1] https://review.openstack.org/#/c/229351/
[2] https://review.openstack.org/#/c/280099/
[3] https://blueprints.launchpad.net/nova/+spec/hyper-v-remotefx
[4] https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg04201.html
[5] https://git.openstack.org/cgit/openstack/nova-specs/tree/specs/newton/approved/virt-device-role-tagging.rst

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160606/38a1b357/attachment.html>