<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:none;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-AU" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Hi Cyborg Team!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Karl from Helm Team.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">When creating a VM with the correct flavor, the mdev gets created by cyborg agent and I can see it in the nodedev-list --cap mdev.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">However Nova then fails with:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">nova.virt.libvirt.driver [<removed>- - default default] Searching for available mdevs... _get_existing_mdevs_not_assigned /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">:8357<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">2023-09-21 14:34:47.808 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Available mdevs at: set().<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] No available mdevs where found. Creating an new one... _allocate_mdevs
/var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driv<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">er.py:8496<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] Attempting to create new mdev... _create_new_mediated_device /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py:8385<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">2023-09-21 14:34:48.455 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Failed to create mdev. No free space found among the following devices:
['pci_0000_4b_03_1', … <truncated list>].<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">2023-09-21 14:34:48.456 1901814 ERROR nova.compute.manager [<removed> - - default default] [instance: 2026e2a2-b17a-43ab-adcb-62a907f58b51] Instance failed to spawn:
nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: mdev-capable resource is not available.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Once this happened, ARQ removes the mdev and cleans up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">I’ve got Cyborg 2023.2 running and have a device profile like so:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">karl@Karls-Air ~ % openstack accelerator device profile show e2b07e11-fe69-4f33-83fc-0f9e38adb7ae<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">+-------------+---------------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| Field | Value |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">+-------------+---------------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| created_at | 2023-09-21 13:30:05+00:00 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| updated_at | None |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| uuid | e2b07e11-fe69-4f33-83fc-0f9e38adb7ae |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| name | VGPU_A40-Q48 |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| groups | [{'resources:VGPU': '1', 'trait:CUSTOM_NVIDIA_2235_A40_48Q': 'required'}] |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| description | None |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">+-------------+---------------------------------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">karl@Karls-Air ~ %<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">I can see the allocation candidate:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">karl@Karls-Air ~ % openstack allocation candidate list --resource VGPU=1 | grep A40<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">| 41 | VGPU=1 | 229bf15f-5689-3d2c-b37b-5c8439ea6a71 | VGPU=0/1 | OWNER_CYBORG,CUSTOM_NVIDIA_2235_A40_48Q |<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">karl@Karls-Air ~ %<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Am I missing something critical here? Because I cannot seem to figure this out… have I got a PCI address wrong, or something?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Any help from the Cyborg or Nova teams would be fantastic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1D1D1D;mso-ligatures:none;mso-fareast-language:EN-GB">Thanks,<br>
Karl.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>