On 14/08/2025 14:34, Dan Smith wrote:
I think this is an example of nova never intended to support the collocation of VMs but never prevented either. Never prevented it for sure, but we have told people time and again not to do it on principle. I believe it has always been the design that Nova assumes full control of the node and previous attempts to avoid things like reaping unknown VMs as deleted were purely to prevent accidental data loss.
its a little more subtl then that. when i first started contibuting to nova i had a very similar converstaion with many different cores about this. you are correct that we have always discouraged this type of colocation. it was howevr coniserd "allowed" i don't want to say supported since we never tested it but thing like contail vrouter i belive were first releases as VNFs that ran in a vm to provide networkign to the other nova vms. this was in the hevana/icehouse time frame so i have lost all memory of the details. you are very much correct that this only really came up after that in the context the perodic to reeap runing instnace that we did not belive hsoudl be on the host. we had extensive conversations on if checking the libvirt metadata for the nova xml namespace metaddadta was enough to determin if it was a nova managed vms. the up shot of that was yes in theory but no we did nto want to commit to that.
And therefore users out there started to rely on this capability. Now we started breaking that capability with the compute_uuid change. But we don't necessarily need to break it with that change. We could have both the compute_uuid change verifying that no nova VMs are running on the host and therefore preventing host rename, and still keeping the old behavior of allowing non nova VMs to run on the host. We would simply need to explicitly check the VMs reported by libvirt if they have nova metadata or not. I'm not sure what we would lose with this simple change. It's certainly true that we could do this, but IMHO it would also open a can of worms in the form of blessing the idea that we allow other VMs alongside our instances. If that regresses in the future, we'll be expected to fix it. If other requests to change behavior for non-Nova instances come along, the same assumption-to-fix may be made.
the rules for doing this in the past were alwsy efectivly the follwoing you must adjust https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.res... https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.res... https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.res... and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.res... to account for the usage of the other vms. in general you shoudl avoid usign https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.res... and prefer https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vcp... or the newer https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu... and https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu... to defien which cores nova can use for its vms. you must not ever uses cores in the *_set config option for non nova managed vms. you many not use any device allow by https://docs.openstack.org/nova/latest/configuration/config.html#pci.device_... for non nova managed vm, that all appleis to the generic mdev feature or device passhtough in general like pmem. i.e. all devices manged by nova shoudl be exlcitlvy manage by it. we have alwasy said while this should be possibel to do its is not supproted by the upstream project offically and we would not build out new feature to make this more supported in the future. all of the above has only ever been true for the libvirt and xen as far as i am aware although i have only ever seen this done with libvirt.
I'm not -2 but my preference would be to not open ourselves to that future.
im not opposed to streating our statement about this to say its explicitly not support and you do so at your own risk. that is effectively the sentiment of what has always been communicated in this regard. the last time i had more then a trowaway conversation about this was with cdent at the most recent denver or vancouver ptg in the context of support Kubernetes and or zun on the same comptue node as nova. effectivly woudl it be possibel to deploy both at the same tiem and partion the host resouce so some could be used for conteiners or something like kubevirt (i dont think that existed then) via kubernetes or zun, and the rest could be used for nova. we dicssed these config option in some detail but i think the over all recomemation was its bettwer to just have a pool of baremental host in ironic an use some for k8s and some for nova as needed. i.e. partion on the host level rahter then within the host level. there was some interest in this general idea by folks that wanted to run opesntack on kubernetese as well. i do tend to agree that while it is possible to do it was never really intended to be something we developed in the future.
--Dan