I think this is an example of nova never intended to support the collocation of VMs but never prevented either.
Never prevented it for sure, but we have told people time and again not to do it on principle. I believe it has always been the design that Nova assumes full control of the node and previous attempts to avoid things like reaping unknown VMs as deleted were purely to prevent accidental data loss.
And therefore users out there started to rely on this capability. Now we started breaking that capability with the compute_uuid change. But we don't necessarily need to break it with that change. We could have both the compute_uuid change verifying that no nova VMs are running on the host and therefore preventing host rename, and still keeping the old behavior of allowing non nova VMs to run on the host. We would simply need to explicitly check the VMs reported by libvirt if they have nova metadata or not. I'm not sure what we would lose with this simple change.
It's certainly true that we could do this, but IMHO it would also open a can of worms in the form of blessing the idea that we allow other VMs alongside our instances. If that regresses in the future, we'll be expected to fix it. If other requests to change behavior for non-Nova instances come along, the same assumption-to-fix may be made. I'm not -2 but my preference would be to not open ourselves to that future. --Dan