<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mer. 21 juin 2023 à 18:23, Dmitriy Rabotyagov <<a href="mailto:noonedeadpunk@gmail.com" target="_blank">noonedeadpunk@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I can recall in quite recent release notes in Nvidia drivers, that now they do allow attaching multiple vGPUs to a single VM, but I can recall Sylvain said that is not exactly as it sounds like and there're severe limitations to this advertised feature.<div dir="auto"><br></div></div></blockquote><div><br></div><div><div class="gmail_quote"><div>That's the problem with this feature
enablement in Nova : we mostly depend on a very specific external Linux
driver. So, tbc, if you want to use vGPU, please rather look at the
Nvidia documentation *before* :)</div><div>About multiple vGPUs, Nvidia says it depends on the GPU architecture (and that was changing since the last years) :<br></div><div><br></div><div>(quoting Nvidia here)<br></div><div><i>The supported vGPUs depend on
the architecture of the GPU on which the vGPUs reside:
</i></div><div><ul><li><i>For GPUs based on the NVIDIA Volta architecture and later GPU architectures,
<b>all</b> Q-series and C-series vGPUs are supported. On GPUs that support the
Multi-Instance GPU (MIG) feature, both time-sliced and MIG-backed vGPUs are
supported.
</i></li><li><i>For GPUs based on the NVIDIA Pascal™ architecture, only Q-series
and C-series vGPUs that are allocated all of the physical GPU's frame buffer are
supported.
</i></li><li><i>For GPUs based on the NVIDIA NVIDIA Maxwell™ graphic architecture,
only Q-series vGPUs that are allocated all of the physical GPU's frame buffer are
supported.
</i></li></ul></div><div><div>
<p><i>You can assign multiple vGPUs with differing amounts of frame buffer to a single VM,
provided the board type and the series of all the vGPUs is the same. For example, you can
assign an A40-48C vGPU and an A40-16C vGPU to the same VM. However, you cannot assign an
A30-8C vGPU and an A16-8C vGPU to the same VM. <br></i></p><div><a href="https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-red-hat-el-kvm/index.html#multiple-vgpu-support-vgpus" target="_blank">https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-red-hat-el-kvm/index.html#multiple-vgpu-support-vgpus</a></div>
</div></div><div>As a reminder, you can find the vGPU types here <a href="https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#virtual-gpu-types-grid-reference" target="_blank">https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#virtual-gpu-types-grid-reference</a></div><div>Basically,
what changed is that with the latest Volta and Ampere architecture,
Nvidia was able to provide different vGPUs with sliced frame buffer
recently, while previously Nvidia was only able to pin a vGPU taking the
whole pGPU frame buffer to a single VM, which was actually limiting de
facto the instance to only have one single vGPU attached (or having a
second vGPU attached from another pGPU, which is non trivial to
schedule)</div><div><br></div>For that reason, we initially limited the
VGPU allocation requests to a maximum of 1 in Nova since it was horribly
depending on hardware, but I eventually tried to propose to remove that
limitation with <a href="https://review.opendev.org/c/openstack/nova/+/845757" target="_blank">https://review.opendev.org/c/openstack/nova/+/845757</a>
which would need some further work and testing (which is nearly
impossible with upstream CI since the nvidia drivers are proprietary and
licensed).<br></div><div class="gmail_quote">Some operator wanting to
lift that current limitation would get all my attention if he/she would
volunteer for *testing* such patch. Ping me on IRC #openstack-nova
(bauzas) and we could proceed quickly.</div><div class="gmail_quote"></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"></div><div dir="auto">Also I think in MIG mode it's possible to split GPU in a subset of supported (but different) flavors, though I have close to no idea how scheduling would be done in this case.</div></div><br></blockquote><div><br></div><div><div>This is quite simple : you need to create different MIG instances
using different heterogenous profiles and you'll see then that *some*
mdev types will accordingly have an inventory of 1.</div><div>You could
then use some new feature we introduced in Xena, which allows the nova
libvirt driver to create different custom resource classes : <a href="https://specs.openstack.org/openstack/nova-specs/specs/xena/implemented/generic-mdevs.html" target="_blank">https://specs.openstack.org/openstack/nova-specs/specs/xena/implemented/generic-mdevs.html</a> <br></div><div><br></div><div>Again,
testing this on real production is the crux of the problem. We provided
as many functional tests as we were able in order to verify such
things, but getting a real MIG-backed GPU and setting the confs
appropriately is something we are missing and which would be useful for
tracking bugs.</div><div><br></div><div>Last point, I'm more than open
to collaborating with CERN or any other operator wanting to stabilize
the vGPU feature enablement in Nova. I know that the existing feature
presents a quite long list of bug reports and has some severe
limitations, but I'd be more happy with having some guidance from the
operators on how and what to stabilize.</div><font color="#888888"><div><br></div><div>-Sylvain</div><div><br></div></font> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 21, 2023, 17:36 Ulrich Schwickerath <<a href="mailto:ulrich.schwickerath@cern.ch" target="_blank">ulrich.schwickerath@cern.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi, again,</p>
<p>here's a link to my slides:<br>
</p>
<p><a href="https://cernbox.cern.ch/s/v3YCyJjrZZv55H2" rel="noreferrer" target="_blank">https://cernbox.cern.ch/s/v3YCyJjrZZv55H2</a></p>
<p>Let me know if it works.</p>
<p>Cheers, Ulrich</p>
</div></blockquote></div></blockquote><div><br></div><div> </div></div></div>