<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi, all,</p>
<p>just to add to the discussion, at CERN we have recently deployed
a bunch of A100 GPUs in PCI passthrough mode, and are now looking
into improving their usage by using MIG. From the NOVA point of
view things seem to work OK, we can schedule VMs requesting a
VGPU, the client starts up and gets a license token from our
NVIDIA license server (distributing license keys is our private
cloud is relatively easy in our case). It's a PoC only for the
time being, and we're not ready to put that forward as we're
facing issues with CUDA on the client (it fails immediately in
memory operations with 'not supported', still investigating why
this happens). <br>
</p>
<p>Once we get that working it would be nice to be able to have a
more fine grained scheduling so that people can ask for MIG
devices of different size. The other challenge is how to set
limits on GPU resources. Once the above issues have been sorted
out we may want to look into cyborg as well thus we are quite
interested in first experiences with this.</p>
<p>Kind regards, </p>
<p>Ulrich<br>
</p>
<div class="moz-cite-prefix">On 13.01.23 21:06, Dmitriy Rabotyagov
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAPd_6Av5VPp+6Co488X2+GFYAv-4S_m=B34NZR01nJwaoDNLOw@mail.gmail.com">
<div dir="auto">
<div>To have that said, deb/rpm packages they are providing
doesn't help much, as:
<div dir="auto">* There is no repo for them, so you need to
download them manually from enterprise portal</div>
<div dir="auto">* They can't be upgraded anyway, as driver
version is part of the package name. And each package
conflicts with any another one. So you need to explicitly
remove old package and only then install new one. And yes,
you must stop all VMs before upgrading driver and no, you
can't live migrate GPU mdev devices due to that now being
implemented in qemu. So deb/rpm/generic driver doesn't
matter at the end tbh.</div>
<br>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">пт, 13 янв. 2023 г., 20:56
Cedric <<a href="mailto:yipikai7@gmail.com" moz-do-not-send="true" class="moz-txt-link-freetext">yipikai7@gmail.com</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto"><br>
Ended up with the very same conclusions than Dimitry
regarding the use of Nvidia Vgrid for the VGPU use case
with Nova, it works pretty well but:<br>
<br>
- respecting the licensing model as operationnal
constraints, note that guests need to reach a license
server in order to get a token (could be via the Nvidia
SaaS service or on-prem)<br>
- drivers for both guest and hypervisor are not easy to
implement and maintain on large scale. A year ago,
hypervisors drivers were not packaged to Debian/Ubuntu,
but builded though a bash script, thus requiering
additional automatisation work and careful attention
regarding kernel update/reboot of Nova hypervisors.<br>
<br>
Cheers</div>
<br>
<br>
On Fri, Jan 13, 2023 at 4:21 PM Dmitriy Rabotyagov <<a href="mailto:noonedeadpunk@gmail.com" rel="noreferrer
noreferrer noreferrer noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">noonedeadpunk@gmail.com</a>>
wrote:<br>
><br>
> You are saying that, like Nvidia GRID drivers are
open-sourced while<br>
> in fact they're super far from being that. In order
to download<br>
> drivers not only for hypervisors, but also for guest
VMs you need to<br>
> have an account in their Enterprise Portal. It took
me roughly 6 weeks<br>
> of discussions with hardware vendors and Nvidia
support to get a<br>
> proper account there. And that happened only after
applying for their<br>
> Partner Network (NPN).<br>
> That still doesn't solve the issue of how to provide
drivers to<br>
> guests, except pre-build a series of images with
these drivers<br>
> pre-installed (we ended up with making a DIB element
for that [1]).<br>
> Not saying about the need to distribute license
tokens for guests and<br>
> the whole mess with compatibility between hypervisor
and guest drivers<br>
> (as guest driver can't be newer then host one, and
HVs can't be too<br>
> new either).<br>
><br>
> It's not that I'm protecting AMD, but just saying
that Nvidia is not<br>
> that straightforward either, and at least on paper
AMD vGPUs look<br>
> easier both for operators and end-users.<br>
><br>
> [1] <a href="https://github.com/citynetwork/dib-elements/tree/main/nvgrid" rel="noreferrer noreferrer noreferrer noreferrer
noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/citynetwork/dib-elements/tree/main/nvgrid</a><br>
><br>
> ><br>
> > As for AMD cards, AMD stated that some of their
MI series card supports SR-IOV for vGPUs. However, those
drivers are never open source or provided closed source to
public, only large cloud providers are able to get them.
So I don't really recommend getting AMD cards for vGPU
unless you are able to get support from them.<br>
> ><br>
><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</body>
</html>