sorry for stepping in so late. We have seen a very similar issue to https://bugs.launchpad.net/nova/+bug/2015892 and the one described here. Our case is maybe a bit different: we have a box with 4 A100 cards, and we wanted to set it up with MIG enabled on all of them. What we've seen was that more mediated devices were created than actually available, and that (both nova and cyborg) created too many resource providers in placement. Then, depending on which resource provider was selected, the VM failed to spawn because the selected devices was not available. We've now a patch for Nova for this which seems to work for our use case. The basic idea of the patch is to check how many instances are available for each of the physical GPUs (from the description field of one of the devices, e.g. /sys/class/mdev_bus/0000\:81\:00.4/mdev_supported_types/nvidia-476/description) and ensure that only this number of resource providers is created in placement. A side effect is that it also works if the GPUs in the hypervisor have different MIG setups (with the restriction that each card must only be setup with a single MIG type e.g. 2g.10gb,2g.10gb,2g.10gb for GPU0 and 3g.20gb,3g.20gb for GPU1 etc). It's not perfect, and in particular I have not tested if this breaks the regular VGPU setup for time based slicing of the GPUs nor how to do this with Cyborg. The VGPU type selection is done via a trait and a custom resource. If you're interested, feel free to get back to me.

Cheers, Ulrich

On 22/09/2023 07:49, Alex Song (宋文平) wrote:


Please reference the nvidia official doc: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#introduction



发件人: Karl Kloppenborg [mailto:kkloppenborg@resetdata.com.au]
发送时间: 2023922 10:53
收件人: Alex Song (宋文平) <songwenping@inspur.com>; sbauza@redhat.com
抄送: openstack-discuss@lists.openstack.org
主题: Re: Cyborg nova reports mdev-capable resource is not available


Ah thank you for pointing me towards that Alex.


I guess, I should probably look at the MIG pathway.

I wonder if it’s possible to do vGPU profiles in MIG configuration.


Have you any experience with this?




From: Alex Song (宋文平) <songwenping@inspur.com>
Date: Friday, 22 September 2023 at 12:17 pm
To: Karl Kloppenborg <
kkloppenborg@resetdata.com.au>, sbauza@redhat.com <sbauza@redhat.com>
openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>
答复: Cyborg nova reports mdev-capable resource is not available


Hi Karl,

         Your problem is similar with the bug: https://bugs.launchpad.net/nova/+bug/2015892

         I guess you don’t split the mig if using A serial card.


发件人: Karl Kloppenborg [mailto:kkloppenborg@resetdata.com.au]
发送时间: 2023922 0:43
收件人: Sylvain Bauza <sbauza@redhat.com>
抄送: openstack-discuss@lists.openstack.org
主题: Re: Cyborg nova reports mdev-capable resource is not available


Hi Sylvian,


Thanks for getting back to me.

So the vGPU is available and cyborg is allocating it using ARQ binding.

You can see Nova receives this request:


2023-09-21 16:38:51.889 1901814 DEBUG nova.compute.manager [None req-97062e9c-0c44-480e-9918-4a5a810175b2 78e83e5a446e4071ae43e823135dcb3c 21eb701c2a1f48b38dab8f34c0a20902 - - default default] ARQs for spec:{'2d60c353-0419-4b67-8cb7-913fc6f5cef9': {'uuid': '2d60c353-0419-4b67-8cb7-913fc6f5cef9', 'state': 'Bound', 'device_profile_name': 'VGPU_A40-Q48', 'device_profile_group_id': 0, 'hostname': 'gpu-c-01', 'device_rp_uuid': '229bf15f-5689-3d2c-b37b-5c8439ea6a71', 'instance_uuid': '1b090007-791b-4997-af89-0feb886cf11d', 'project_id': None, 'attach_handle_type': 'MDEV', 'attach_handle_uuid': '866bd6a5-b156-4251-a969-64fefb32f16f', 'attach_handle_info': {'asked_type': 'nvidia-566', 'bus': 'ca', 'device': '01', 'domain': '0000', 'function': '1', 'vgpu_mark': 'nvidia-566_0'}, 'links': [{'href': 'http://cyborg-api.openstack.svc.cluster.local:6666/accelerator/v2/accelerator_requests/2d60c353-0419-4b67-8cb7-913fc6f5cef9', 'rel': 'self'}], 'created_at': '2023-09-21T16:38:42+00:00', 'updated_at': '2023-09-21T16:38:42+00:00'}}, ARQs for network:{} _build_resources /var/lib/openstack/lib/python3.10/site-packages/nova/compute/manager.py:2680


So the mdev is then allocated in the resource providers at that point.


Is there some cyborg nova patching code I am missing?





From: Sylvain Bauza <sbauza@redhat.com>
Date: Friday, 22 September 2023 at 1:49 am
To: Karl Kloppenborg <
openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>
Subject: Re: Cyborg nova reports mdev-capable resource is not available



Le jeu. 21 sept. 2023 à 17:27, Karl Kloppenborg <kkloppenborg@resetdata.com.au> a écrit :

Hi Cyborg Team!

Karl from Helm Team.


When creating a VM with the correct flavor, the mdev gets created by cyborg agent and I can see it in the nodedev-list --cap mdev.

However Nova then fails with:

nova.virt.libvirt.driver [<removed>- - default default] Searching for available mdevs... _get_existing_mdevs_not_assigned /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py


2023-09-21 14:34:47.808 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Available mdevs at: set().

2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] No available mdevs where found. Creating an new one... _allocate_mdevs /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driv


2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] Attempting to create new mdev... _create_new_mediated_device /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py:8385

2023-09-21 14:34:48.455 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Failed to create mdev. No free space found among the following devices: ['pci_0000_4b_03_1', … <truncated list>].

2023-09-21 14:34:48.456 1901814 ERROR nova.compute.manager [<removed> - - default default] [instance: 2026e2a2-b17a-43ab-adcb-62a907f58b51] Instance failed to spawn: nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: mdev-capable resource is not available.



I don't exactly remember how Cyborg passes the devices to nova/libvirt but this exception is because none of the available GPUs have either existing mdevs or capability for creating mdevs.

You should first check sysfs to double-check the state of our GPU devices in order to understand how much of vGPU capacity you still have.




Once this happened, ARQ removes the mdev and cleans up.


I’ve got Cyborg 2023.2 running and have a device profile like so:

karl@Karls-Air ~ % openstack accelerator device profile show e2b07e11-fe69-4f33-83fc-0f9e38adb7ae


| Field       | Value                                                                     |


| created_at  | 2023-09-21 13:30:05+00:00                                                 |

| updated_at  | None                                                                      |

| uuid        | e2b07e11-fe69-4f33-83fc-0f9e38adb7ae                                      |

| name        | VGPU_A40-Q48                                                              |

| groups      | [{'resources:VGPU': '1', 'trait:CUSTOM_NVIDIA_2235_A40_48Q': 'required'}] |

| description | None                                                                      |


karl@Karls-Air ~ %


I can see the allocation candidate:

karl@Karls-Air ~ % openstack allocation candidate list --resource VGPU=1 | grep A40

|  41 | VGPU=1     | 229bf15f-5689-3d2c-b37b-5c8439ea6a71 | VGPU=0/1                | OWNER_CYBORG,CUSTOM_NVIDIA_2235_A40_48Q |

karl@Karls-Air ~ %



Am I missing something critical here? Because I cannot seem to figure this out… have I got a PCI address wrong, or something?


Any help from the Cyborg or Nova teams would be fantastic.