Hello Andrew!
I wonder if it has to do with how the volumes/storage are set up with your instance. If the instance is not an ephemeral instance, I think the behavior you describe is what you could see. It looks like according to this document ( https://docs.openstack.org/nova/latest/contributor/evacuate-vs-rebuild.html#...) on evacuation vs rebuilding, volume-backed instances don't actually rebuild with different images. If I've misread that, I welcome the correction.
What does the storage configuration for your virtual machines (that is, boot disk, volumes, images, etc.) look like? According to the API docs ( https://docs.openstack.org/api-ref/compute/#rebuild-server-rebuild-action) you do get different behaviors if the instance is volume-backed or not.
Thanks!
Jadon
On Tue, Jul 16, 2024 at 2:46 PM Andrew Bogott abogott@wikimedia.org wrote:
I'm puzzled by nova instance rebuilds.
When I run 'openstack server rebuild --image <whatever>' on a new test VM, it does what I expect:
throws away the old server image
boots up a brand new server using the requested image with the old
user/vendor data
- attaches new server to the old network port and cinder volumes.
That's just what I want! Alas, when I try this with the actual old VMs in need of update, usually it does a totally different thing:
reboots the server
changes the nova db record to reflect the requested VM (so that the
server /looks/ like it was rebuilt in 'openstack server show')
- That's it!
It absolutely does not rebuild the actual server disk image. So, for instance, any files that were present in '/' of the old server are still present post-rebuild. lsb_release still shows the OS of the original base image, disagreeing with the the OS version displayed by 'openstack server show'.
I've been reading logfiles for days but I don't see nova ever complain (in either case) that it's doing something unexpected and in both cases the servers wind up in state 'Active'.
Am I misunderstanding what this is supposed to do?
If not, does anyone have thoughts about what might be going wrong? I assume nova is encountering a roadbump during the rebuild and doing a graceful revert, but if so it sure isn't telling me.
Bobcat, kvm, VM images are hosted on ceph/rbd.