<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hello,
<div><br>
</div>
<div>We are using vGPUs with Nova on OpenStack Xena release and we’ve had a fairly good experience integration</div>
<div>NVIDIA A10 GPUs into our cloud.</div>
<div><br>
</div>
<div>As we see it there is some painpoints that just goes with mantaining the GPU feature.</div>
<div><br>
</div>
<div>- There is a very tight coupling of the NVIDIA driver in the guest (instance) and on the compute node that needs to</div>
<div> be managed.</div>
<div><br>
</div>
<div>- Doing maintainance need more planning i.e powering off instances, NVIDIA driver on compute node needs to be</div>
<div> rebuilt on hypervisor if kernel is upgraded unless you’ve implemented DKMS for that.</div>
<div><br>
</div>
<div>- Because we’ve different flavor of GPU (we split the A10 cards into different flavors for maximum utilization of</div>
<div> other compute resources) we added custom traits in the Placement service to handle that, handling that with</div>
<div> a script since doing anything manually related to GPUs you will get confused quickly. [1]</div>
<div><br>
</div>
<div>- Since Nova does not handle recreation of mdevs (or use the new libvirt autostart feature for mdevs) we have</div>
<div> a systemd unit that executes before the nova-compute service that walks all the libvirt domains and does lookups</div>
<div> in Placement to recreate the mdevs before nova-compute start. [2] [3] [4]</div>
<div><br>
</div>
<div>Best regards</div>
<div>Tobias</div>
<div><br>
</div>
<div>DISCLAIMER: Below is provided without any warranty of actually working for you or your setup and does</div>
<div>very specific things that we need and is only provided to give you some insight and help. Use at your own risk.</div>
<div><br>
</div>
<div>[1] <a href="https://paste.opendev.org/show/b6FdfwDHnyJXR0G3XarE/">https://paste.opendev.org/show/b6FdfwDHnyJXR0G3XarE/</a></div>
<div>[2] <a href="https://paste.opendev.org/show/bGtO6aIE519uysvytWv0/">https://paste.opendev.org/show/bGtO6aIE519uysvytWv0/</a></div>
<div>[3] <a href="https://paste.opendev.org/show/bftOEIPxlpLptkosxlL6/">https://paste.opendev.org/show/bftOEIPxlpLptkosxlL6/</a></div>
<div>[4] <a href="https://paste.opendev.org/show/bOYBV6lhRON4ntQKYPkb/">https://paste.opendev.org/show/bOYBV6lhRON4ntQKYPkb/</a></div>
</body>
</html>