Nova support for multiple vGPUs?

Sean Mooney smooney at
Thu Apr 21 10:20:48 UTC 2022

On Wed, 2022-04-20 at 16:42 +0000, Sigurd Kristian Brinch wrote:
> Hi,
> As far as I can tell, libvirt/KVM supports multiple vGPUs per VM
> (,
> but in OpenStack/Nova it is limited to one vGPU per VM
> (
> Is there a reason for this limit?
yes nvidia
> What would be needed to enable multiple vGPUs in Nova?
so you can technically do it today if you have 2 vGPU for seperate physical gpu cards
but nvidia do not support multiple vGPUs form the same card.

nova does not currently provide a way to force the gpu allocation to be from seperate cards.

well thats not quite true you could

you would have to use the named group syntax to request them so instaed of resources:vgpu=2

you woudl do 


the name after resouces_ is arbitray group name provided it conforms to this regex '([a-zA-Z0-9_-]{1,64})?'

we stongly dislike this approch.
first of all using group_policy=isolate is a gloabl thing meaning that no request groups can come form the same provider

that means you can not have to sriov VFs from the same physical nic as a result of setting it.
if you dont set group_policy the default is none which means you no longer are guarenteed that they will come form different providres

so what you woudl need to do is extend placment to support isolating only sepeicic named groups
and then expose that in nova via flavor extra specs which is not particaly good ux as it rather complicated and means you need to
understand how placement works in depth. placement shoudl really be an implemenation detail

that fixes the confilct with sriov and all other usages of resouce groups like bandwith based qos

the slightly better approch wouls be to make this simplere to use by doing somtihng liek this


we would still need the placement feature to isolate by group
but we can hide the detail form the end user with a pre filter in nova
which will transfrom the resouce request and split it up into groups automatically

this is a long way to say that if it was not for limiations in the iommu on nvidia gpus and the fact that they cannot map two vgpus
to from on phsyical gpu to a singel vm this would already work out of hte box wiht just
resources:vgpu=2. perhaps when intel lauch there discret datacenter gpus there vGPU implementaiotn will not have this limiation.
we do not prevent you from requestin 2 vgpus today it will just fail when qemu tries to use them.

we also have not put the effort into working around the limiation in nvidias hardware since ther drivers also used to block this
until the ampear generation and there has nto been a large request to support multipel vgpus form users.

ocationally some will ask about it but in general peopel either do full gpu passthough or use 1 vgpu instance.

hopefully that will help.
you can try the first approch today if you have more then one physical gpu per host

just be aware of the limiation fo group_policy=isolate


> BR
> Sigurd

More information about the openstack-discuss mailing list