Open Stack

On Thu, 2023-04-06 at 13:28 +0200, Sylvain Bauza wrote:
> Le jeu. 6 avr. 2023 à 11:25, Dmitriy Rabotyagov <noonedeadpunk at gmail.com> a
> écrit :
> 
> > I think I just came up with another "usecase" or better say missing
> > functionality. So in case VM is stuck in `unshelving` state, for
> > example due to messaging issues or smth, there's no clean way of
> > recovering VM from this state.
> > Given you will reset state to active - you won't be able to execute
> > `stop` since VM is not assigned to any compute (and fail with
> > "instance not ready"), as it was shelved. So then rebuild could be
> > used, since it will pass VM to be assigned to some host as a result.
> > Another way around would be of course updating the database, setting
> > VM back to `shelved_offloaded` and trying to unshelve again, but I
> > hate messing up with DB.
> > 
> > I think this kinda brings me back to Sean's point of having an API
> > call to re-create a VM while keeping it's data, as that would cover
> > such corner-cases as well.
> > 
> > 
> FWIW, we agreed on the vPTG to add another separate policy for cold-migrate
> (when the 'host' parameter is provided), so you could just modify the
> existing cold-migrate policy (without the 'host' parameter by default) to
> be able to be called by an end-user.
> If so, an user could ask to move their instances and restart them by this,
> and if they see some problem, they could then revert the migration.
> 
> I should be working on it by the next weeks hopefully.

that would help as long as the vm is not stuck in an error state.
i.e. it wont help in the stuck in unshelving case.
but it would help in the noisy neighbour case potentially.
> -Sylvain
> 
> вт, 21 мар. 2023 г. в 15:59, Dan Smith <dms at danplanet.com>:
> > > 
> > > > Basically they have an additional and unusual compute host recovery
> > > > process, where a compute host after a failure is brought back by the
> > > > same name. Then they rebuild the servers on the same compute host
> > > > where the servers were running before. When the server's disk was
> > > > backed by a volume, so its content was not lost by the compute host
> > > > failure, they don't want to lose it either in the recovery process.
> > > > The evacute operation clearly would be a better fit to do this, but
> > > > that disallows evacuating to the "same" host. For a long time rebuild
> > > > just allowed "evacuating to the same host". So they went with it.
> > > 
> > > Aside from the "should this be possible" question, is rebuild even
> > > required in this case? For the non-volume-backed instances, we need
> > > rebuild to re-download the image and create the root disk. If it's
> > > really required for volume-backed instances, I'm guessing there's just
> > > some trivial amount of state that isn't in place on recovery that the
> > > rebuild "solves". It is indeed a very odd fringe use-case that is an
> > > obvious mis-use of the function.
> > > 
> > > > At the moment I did not find a prohibition in the documentation to
> > > > bring back a failed compute host by the same name. If I missed it or
> > > > this is not recommended for any reason, please let me know.
> > > 
> > > I'm not sure why this would be specifically documented, but since
> > > compute nodes are not fully stateless, your scenario is basically
> > > "delete part of the state of the system and expect things to keep
> > > working" which I don't think is reasonable (nor something we should need
> > > to document).
> > > 
> > > Your scenario is basically the same as one where your /var/lib/nova is
> > > mounted on a disk that doesn't come up after reboot, or on NFS that was
> > > unavailable at boot. If nova were to say "meh, a bunch of state
> > > disappeared, I must be a rebuilt compute host" then it would potentially
> > > destroy (or desynchronize) actual state in other nodes (i.e. the
> > > database) for a transient/accidental situation. TBH, we might should
> > > even explicitly *block* rebuild on an instance that appears to be
> > > missing its on-disk state to avoid users, who don't know the state of the
> > > infra, from doing this to try to unblock their instances while ops are
> > > doing maintenance.
> > > 
> > > I will point out that bringing back a compute node under the same name
> > > (without cleaning the residue first) is strikingly similar to renaming a
> > > compute host, which we do *not* support. As of Antelope, the compute
> > > node would detect your scenario as a potential rename and refuse to
> > > start, again because of state that has been lost in the system. So just
> > > FYI that an actual blocker to your scenario is coming :)
> > > 
> > > > Clearly in many clouds evacuating can fully replace what they do here.
> > > > I believe they may have chosen this unusual compute host recovery
> > > > option to have some kind of recovery process for very small
> > > > deployments, where you don't always have space to evacuate before you
> > > > rebuilt the failed compute host. And this collided with a deployment
> > > > system which reuses host names.
> > > > 
> > > > At this point I'm not sure if this really belongs to the rebuild
> > > > operation. Could easily be better addressed in evacuate. Or in the
> > > > deployment system not reusing hostnames.
> > > 
> > > Evacuate can't work for this case either because it requires the compute
> > > node to be down to perform. As you note, bringing it back under a
> > > different name would solve that problem. However, neither "evacuate to
> > > same host" or "use rebuild for this recovery procedure" are reasonable,
> > > IMHO.
> > > 
> > > --Dan
> > > 
> > 
> >