Open Stack

Hi all,

Currently openstack support vGPU as follow:
https://docs.openstack.org/nova/latest/admin/virtual-gpu.html

In order to support it, admin should plan ahead and configure the vGPU before deployable as follow:
https://docs.openstack.org/nova/latest/configuration/config.html#devices.enabled_vgpu_types
This is very inconvenient for the administrator, this method has a limitation that a same PCI address does not provide two different types.

Cyborg as an accelerator management tool is more suitable for mdev device management.

One solution as follow：
Firstly, we need a vender driver(this can be a plugin),  it is used to discovery its special devices and report them to placement for schedule.
The difference from the current implementation is that:
1. report the mdev_supported_types as traits to resource provider.
How to discover a GPU type:
$ ls /sys/class/mdev_bus/*/mdev_supported_types
/sys/class/mdev_bus/0000:84:00.0/mdev_supported_types:
nvidia-35  nvidia-36  nvidia-37  nvidia-38  nvidia-39  nvidia-40  nvidia-41  nvidia-42  nvidia-43  nvidia-44  nvidia-45
so here we report nvidia-3*, nvidia-4* as traits to resource provider.
2. Report the number of allocable resources instead of vGPU unit numbers to resource provider inventory
Example for the NVidia V100 PCIe card (one GPU per board) :
Virtual GPU Type     Frame Buffer (Gbytes)      Maximum vGPUs per GPU        Maximum vGPUs per Board
V100D-32Q               32                        1                            1
V100D-16Q               16                        2                            2
V100D-8Q               8                         4                            4
V100D-4Q               4                          8                            8
V100D-2Q               2                         16                           16
V100D-1Q               1                         32                           32
so here we report 32G Buffers(an example, maybe other resources) to resource provider inventory
3. driver should also support a function to create certain mdev type, such as (V100D-1Q,  V100D-2Q,)
Secondly, we need a mdev extend ARQ(it can be a plugin):
Here is an example for fpga ext arq:
https://review.opendev.org/#/c/681005/26/cyborg/objects/extarq/fpga_ext_arq.py@206
The difference is that, we replace the _do_programming to _do_create_mdev
For _do_programming, it is used to create a new FPGA function.
For _do_create_mdev, it is used to create a new type mdev, it will call the implementation function in vendor driver.

At last we need to support a mdev handler for xml generation in nova, we can refer to the cyborg PCI handler in nova

So after the above changes：
Admin can create different SLA devices profiles such as:
{“name”: “Gold_vGPU”,
       “groups”: [
{“resources:vGPU_BUFFERS”: “16”,
        “traits: V100D-16Q,”: “required”,
              }]
}
And
{“name”: “Iron_vGPU”,
       “groups”: [
{“resources:vGPU_BUFFERS”: “1”,
        “traits: V100D-1Q,”: “required”,
              }]
}
Then a tenant can use Gold_vGPU to create with a VM with V100D-16Q vGPU
And another tenant can use Iron_vGPU to create with a VM with V100D-1Q vGPU

When ARQ binding during the VM creating, the Cyborg will call the vendor driver to create expected mdev vGPU.
And these 2 mdev vGPU can be on same physical GPU card.

The mdev extend ARQ and vendor driver can be plugin, they are loose couple with the upstream code.
So the downstream can get the upstream code to customize the own mdev extend ARQ and vendor driver.
Here vGPU is just an example, it can be other mdev devices.


BR
Shaohe Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200623/524d2fe9/attachment-0001.html>