Open Stack

Le jeu. 6 avr. 2023 à 11:25, Dmitriy Rabotyagov <noonedeadpunk at gmail.com> a
écrit :

> I think I just came up with another "usecase" or better say missing
> functionality. So in case VM is stuck in `unshelving` state, for
> example due to messaging issues or smth, there's no clean way of
> recovering VM from this state.
> Given you will reset state to active - you won't be able to execute
> `stop` since VM is not assigned to any compute (and fail with
> "instance not ready"), as it was shelved. So then rebuild could be
> used, since it will pass VM to be assigned to some host as a result.
> Another way around would be of course updating the database, setting
> VM back to `shelved_offloaded` and trying to unshelve again, but I
> hate messing up with DB.
>
> I think this kinda brings me back to Sean's point of having an API
> call to re-create a VM while keeping it's data, as that would cover
> such corner-cases as well.
>
>
FWIW, we agreed on the vPTG to add another separate policy for cold-migrate
(when the 'host' parameter is provided), so you could just modify the
existing cold-migrate policy (without the 'host' parameter by default) to
be able to be called by an end-user.
If so, an user could ask to move their instances and restart them by this,
and if they see some problem, they could then revert the migration.

I should be working on it by the next weeks hopefully.
-Sylvain

вт, 21 мар. 2023 г. в 15:59, Dan Smith <dms at danplanet.com>:
> >
> > > Basically they have an additional and unusual compute host recovery
> > > process, where a compute host after a failure is brought back by the
> > > same name. Then they rebuild the servers on the same compute host
> > > where the servers were running before. When the server's disk was
> > > backed by a volume, so its content was not lost by the compute host
> > > failure, they don't want to lose it either in the recovery process.
> > > The evacute operation clearly would be a better fit to do this, but
> > > that disallows evacuating to the "same" host. For a long time rebuild
> > > just allowed "evacuating to the same host". So they went with it.
> >
> > Aside from the "should this be possible" question, is rebuild even
> > required in this case? For the non-volume-backed instances, we need
> > rebuild to re-download the image and create the root disk. If it's
> > really required for volume-backed instances, I'm guessing there's just
> > some trivial amount of state that isn't in place on recovery that the
> > rebuild "solves". It is indeed a very odd fringe use-case that is an
> > obvious mis-use of the function.
> >
> > > At the moment I did not find a prohibition in the documentation to
> > > bring back a failed compute host by the same name. If I missed it or
> > > this is not recommended for any reason, please let me know.
> >
> > I'm not sure why this would be specifically documented, but since
> > compute nodes are not fully stateless, your scenario is basically
> > "delete part of the state of the system and expect things to keep
> > working" which I don't think is reasonable (nor something we should need
> > to document).
> >
> > Your scenario is basically the same as one where your /var/lib/nova is
> > mounted on a disk that doesn't come up after reboot, or on NFS that was
> > unavailable at boot. If nova were to say "meh, a bunch of state
> > disappeared, I must be a rebuilt compute host" then it would potentially
> > destroy (or desynchronize) actual state in other nodes (i.e. the
> > database) for a transient/accidental situation. TBH, we might should
> > even explicitly *block* rebuild on an instance that appears to be
> > missing its on-disk state to avoid users, who don't know the state of the
> > infra, from doing this to try to unblock their instances while ops are
> > doing maintenance.
> >
> > I will point out that bringing back a compute node under the same name
> > (without cleaning the residue first) is strikingly similar to renaming a
> > compute host, which we do *not* support. As of Antelope, the compute
> > node would detect your scenario as a potential rename and refuse to
> > start, again because of state that has been lost in the system. So just
> > FYI that an actual blocker to your scenario is coming :)
> >
> > > Clearly in many clouds evacuating can fully replace what they do here.
> > > I believe they may have chosen this unusual compute host recovery
> > > option to have some kind of recovery process for very small
> > > deployments, where you don't always have space to evacuate before you
> > > rebuilt the failed compute host. And this collided with a deployment
> > > system which reuses host names.
> > >
> > > At this point I'm not sure if this really belongs to the rebuild
> > > operation. Could easily be better addressed in evacuate. Or in the
> > > deployment system not reusing hostnames.
> >
> > Evacuate can't work for this case either because it requires the compute
> > node to be down to perform. As you note, bringing it back under a
> > different name would solve that problem. However, neither "evacuate to
> > same host" or "use rebuild for this recovery procedure" are reasonable,
> > IMHO.
> >
> > --Dan
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230406/e218fe09/attachment.htm>