<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le jeu. 21 sept. 2023 à 17:27, Karl Kloppenborg <<a href="mailto:kkloppenborg@resetdata.com.au">kkloppenborg@resetdata.com.au</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-535086849548132278">
<div style="overflow-wrap: break-word;" lang="EN-AU">
<div class="m_-7387438457624352011WordSection1">
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Hi Cyborg Team!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Karl from Helm Team.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">When creating a VM with the correct flavor, the mdev gets created by cyborg agent and I can see it in the nodedev-list --cap mdev.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">However Nova then fails with:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">nova.virt.libvirt.driver [<removed>- - default default] Searching for available mdevs... _get_existing_mdevs_not_assigned /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">:8357<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">2023-09-21 14:34:47.808 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Available mdevs at: set().<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] No available mdevs where found. Creating an new one... _allocate_mdevs
/var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driv<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">er.py:8496<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] Attempting to create new mdev... _create_new_mediated_device /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py:8385<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">2023-09-21 14:34:48.455 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Failed to create mdev. No free space found among the following devices:
['pci_0000_4b_03_1', … <truncated list>].<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">2023-09-21 14:34:48.456 1901814 ERROR nova.compute.manager [<removed> - - default default] [instance: 2026e2a2-b17a-43ab-adcb-62a907f58b51] Instance failed to spawn:
nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: mdev-capable resource is not available.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> </span></p></div></div></div></blockquote><div><br></div><div>I don't exactly remember how Cyborg passes the devices to nova/libvirt but this exception is because none of the available GPUs have either existing mdevs or capability for creating mdevs.</div><div>You should first check sysfs to double-check the state of our GPU devices in order to understand how much of vGPU capacity you still have. <br></div><div><br></div><div>-Sylvain</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-535086849548132278"><div style="overflow-wrap: break-word;" lang="EN-AU"><div class="m_-7387438457624352011WordSection1"><p class="MsoNormal"><span style="color:rgb(29,29,29)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Once this happened, ARQ removes the mdev and cleans up.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">I’ve got Cyborg 2023.2 running and have a device profile like so:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">karl@Karls-Air ~ % openstack accelerator device profile show e2b07e11-fe69-4f33-83fc-0f9e38adb7ae<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">+-------------+---------------------------------------------------------------------------+<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| Field | Value |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">+-------------+---------------------------------------------------------------------------+<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| created_at | 2023-09-21 13:30:05+00:00 |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| updated_at | None |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| uuid | e2b07e11-fe69-4f33-83fc-0f9e38adb7ae |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| name | VGPU_A40-Q48 |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| groups | [{'resources:VGPU': '1', 'trait:CUSTOM_NVIDIA_2235_A40_48Q': 'required'}] |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| description | None |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">+-------------+---------------------------------------------------------------------------+<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">karl@Karls-Air ~ %<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">I can see the allocation candidate:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">karl@Karls-Air ~ % openstack allocation candidate list --resource VGPU=1 | grep A40<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">| 41 | VGPU=1 | 229bf15f-5689-3d2c-b37b-5c8439ea6a71 | VGPU=0/1 | OWNER_CYBORG,CUSTOM_NVIDIA_2235_A40_48Q |<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">karl@Karls-Air ~ %<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Am I missing something critical here? Because I cannot seem to figure this out… have I got a PCI address wrong, or something?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Any help from the Cyborg or Nova teams would be fantastic.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(29,29,29)">Thanks,<br>
Karl.<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></blockquote></div></div>