Open Stack

On Mon, 2023-02-06 at 13:15 +0000, Jonathan Rosser wrote:
> Hi Tobias,
> 
> Thankyou for posting the scripts to recreate the mdev, those are very 
> useful and have worked OK in our environment.
> 
> Managing the race conditions between the nvidia gpu manager starting, 
> re-creating the mdev and holding off nova-compute starting until that is 
> all complete seems quite tricky.
assuming you are using systemd you can use the before/after statement to
orcestrate that.

if you can modify either of the systemd service files just create an entirly new one
for the synconisation. make it run After the nvidia-gpu-manager service and before
the nova-compute/libvirt serivce and just invoke /bin/true

see https://www.freedesktop.org/software/systemd/man/systemd.unit.html
for details.

> 
> I see the comments on 
> https://review.opendev.org/c/openstack/nova/+/864418 and I'm also 
> interested to know how the ordering between udev rules execution and the 
> nvidia driver being sufficiently initialised to create mdev can be 
> expressed.
in this case its not really required.
when the PF device is fully initallised that can be used as the trigger for the udev rule to configure the vfs
Alternitivly  you can just use systemd service files.
in general this is out of scope fo the nova docs and we just want to provide some simpel references
that peopl can use but the goal is not to perscibel as specific implemation that will be used.

one approch would be to use somting liek https://github.com/NVIDIA/mig-parted instead and just ensure
that the nova-compute and libvirt service is not executed until after that has completed 

this is very much a case of you should follow the recommendation of the manufacutre of the hardware
and form an upsterem perspeictive we just want to docuemt soemthign that should work for most people
but we dont nessiarly want to fully descipe what to to do.
> 
> Thanks again for the scripts,
> Jonathan.
> 
> On 17/01/2023 08:54, Tobias Urdin wrote:
> > Hello,
> > 
> > We are using vGPUs with Nova on OpenStack Xena release and we’ve had a 
> > fairly good experience integration
> > NVIDIA A10 GPUs into our cloud.
> > 
> > As we see it there is some painpoints that just goes with mantaining 
> > the GPU feature.
> > 
> > - There is a very tight coupling of the NVIDIA driver in the guest 
> > (instance) and on the compute node that needs to
> >   be managed.
> > 
> > - Doing maintainance need more planning i.e powering off instances, 
> > NVIDIA driver on compute node needs to be
> >   rebuilt on hypervisor if kernel is upgraded unless you’ve 
> > implemented DKMS for that.
> > 
> > - Because we’ve different flavor of GPU (we split the A10 cards into 
> > different flavors for maximum utilization of
> >   other compute resources) we added custom traits in the Placement 
> > service to handle that, handling that with
> >   a script since doing anything manually related to GPUs you will get 
> > confused quickly. [1]
> > 
> > - Since Nova does not handle recreation of mdevs (or use the new 
> > libvirt autostart feature for mdevs) we have
> >   a systemd unit that executes before the nova-compute service that 
> > walks all the libvirt domains and does lookups
> >   in Placement to recreate the mdevs before nova-compute start. [2] 
> > [3] [4]
> > 
> > Best regards
> > Tobias
> > 
> > DISCLAIMER: Below is provided without any warranty of actually working 
> > for you or your setup and does
> > very specific things that we need and is only provided to give you 
> > some insight and help. Use at your own risk.
> > 
> > [1] https://paste.opendev.org/show/b6FdfwDHnyJXR0G3XarE/
> > [2] https://paste.opendev.org/show/bGtO6aIE519uysvytWv0/
> > [3] https://paste.opendev.org/show/bftOEIPxlpLptkosxlL6/
> > [4] https://paste.opendev.org/show/bOYBV6lhRON4ntQKYPkb/
>