To be frank I wasn't trying to do hard reboot in that case .Til that it was unobvious for me that it should work or be used at all in such weird scenarios. Like docs say nothing about how hard reboot can be leveraged [1]. So my impression was that hard reboot simply triggers `virsh destroy; virsh create` in the contrary to `virsh reboot` for soft one and that's kind of it, rather than taking care of re-wiring everything.
My biggest problem with DB updates, is that set of users who can execute admin commands towards openstack APIs != set of users who have root access to the hardware or database. So while indeed recovery from such failures could be escalated, at least verifying if the issue is intermittent or not is kinda helpful as well. So having a way to recover from such states via API would be an improvement.
[1] https://docs.openstack.org/nova/latest/user/reboot.html
чт, 6 апр. 2023 г. в 12:56, Sean Mooney smooney@redhat.com:
On Thu, 2023-04-06 at 11:19 +0200, Dmitriy Rabotyagov wrote:
I think I just came up with another "usecase" or better say missing functionality. So in case VM is stuck in `unshelving` state, for example due to messaging issues or smth, there's no clean way of recovering VM from this state.
rebuild would not eb corrct to use there.
Given you will reset state to active
that is not safe to do. the correct fix would be reset it to shevle_offloaded which you currntly would have to do in the db.
- you won't be able to execute
`stop` since VM is not assigned to any compute (and fail with "instance not ready"), as it was shelved. So then rebuild could be used, since it will pass VM to be assigned to some host as a result. Another way around would be of course updating the database, setting VM back to `shelved_offloaded` and trying to unshelve again, but I hate messing up with DB.
I think this kinda brings me back to Sean's point of having an API call to re-create a VM while keeping it's data, as that would cover such corner-cases as well.
well we have talked about allowing reset-state reset to other states in the past or allowing evacuate to work. i proablywoul dnot allow the recreate api to work in that broken state.
the recreate api was not intended for error recovery. it was intented to fullfile two usercases 1.) unify rebuild and resize so you can do either or both from a singel api call. 2.) update your vm so that it gets the latest flavor extra_specs and image properies appleis witout data lose.
вт, 21 мар. 2023 г. в 15:59, Dan Smith dms@danplanet.com:
Basically they have an additional and unusual compute host recovery process, where a compute host after a failure is brought back by the same name. Then they rebuild the servers on the same compute host where the servers were running before. When the server's disk was backed by a volume, so its content was not lost by the compute host failure, they don't want to lose it either in the recovery process. The evacute operation clearly would be a better fit to do this, but that disallows evacuating to the "same" host. For a long time rebuild just allowed "evacuating to the same host". So they went with it.
Aside from the "should this be possible" question, is rebuild even required in this case?
if your vm is boot form voluem or you are using the ceph image backend for nova or nova on nfs then i think all that is requried is hard reboot. there are no port updates/bindings and hard reboot both plugs the netowrk interface into ovs or whatever the backend is on the host but also invokes os-brick to do the same for the volumes. so its not clear to my why rebuild woudl be required in a shared storage case.
For the non-volume-backed instances, we need rebuild to re-download the image and create the root disk.
yes although when you had the hardware failure you could have used evacuate to rebuild the vm on another host. if you could not do that because the vm was pinned to that host then the existing rebuild command is sufficent. if the failure was a motherboard or simialr and the data on disk was not lost then a hard reboot should also be enough for vms with local storage. rebuild would only be required if the data was lost.
If it's really required for volume-backed instances, I'm guessing there's just some trivial amount of state that isn't in place on recovery that the rebuild "solves". It is indeed a very odd fringe use-case that is an obvious mis-use of the function.
ya if hard reboot/power on is not enough i think there is a trival bug there. we are obviouly missing somethign that should be done. power_on/hard reboot are intended to be abel to recreate the vm with its data after the host had been power off and powered on again. so it is ment to do everything required to be able to start the instance. nova has all the info in its database to do that without needing to call the other service like cinder and neutorn.
it woudl be good to know what actully fails if you just do hard reboot and capature that in a bug report.
At the moment I did not find a prohibition in the documentation to bring back a failed compute host by the same name. If I missed it or this is not recommended for any reason, please let me know.
I'm not sure why this would be specifically documented, but since compute nodes are not fully stateless, your scenario is basically "delete part of the state of the system and expect things to keep working" which I don't think is reasonable (nor something we should need to document).
Your scenario is basically the same as one where your /var/lib/nova is mounted on a disk that doesn't come up after reboot, or on NFS that was unavailable at boot. If nova were to say "meh, a bunch of state disappeared, I must be a rebuilt compute host" then it would potentially destroy (or desynchronize) actual state in other nodes (i.e. the database) for a transient/accidental situation. TBH, we might should even explicitly *block* rebuild on an instance that appears to be missing its on-disk state to avoid users, who don't know the state of the infra, from doing this to try to unblock their instances while ops are doing maintenance.
I will point out that bringing back a compute node under the same name (without cleaning the residue first) is strikingly similar to renaming a compute host, which we do *not* support. As of Antelope, the compute node would detect your scenario as a potential rename and refuse to start, again because of state that has been lost in the system. So just FYI that an actual blocker to your scenario is coming :)
Clearly in many clouds evacuating can fully replace what they do here. I believe they may have chosen this unusual compute host recovery option to have some kind of recovery process for very small deployments, where you don't always have space to evacuate before you rebuilt the failed compute host. And this collided with a deployment system which reuses host names.
At this point I'm not sure if this really belongs to the rebuild operation. Could easily be better addressed in evacuate. Or in the deployment system not reusing hostnames.
Evacuate can't work for this case either because it requires the compute node to be down to perform. As you note, bringing it back under a different name would solve that problem. However, neither "evacuate to same host" or "use rebuild for this recovery procedure" are reasonable, IMHO.
--Dan