Open Stack

> My feeling is that we should not attempt to "migrate" any allocations
> or inventories between root or child providers within a compute node,
> period.

While I agree this is the simplest approach, it does put a lot of
responsibility on the operators to do work to sidestep this issue, which
might not even apply to them (and knowing if it does might be
difficult).

> The virt drivers should simply error out of update_provider_tree() if
> there are ANY existing VMs on the host AND the virt driver wishes to
> begin tracking resources with nested providers.
>
> The upgrade operation should look like this:
>
> 1) Upgrade placement
> 2) Upgrade nova-scheduler
> 3) start loop on compute nodes. for each compute node:
>  3a) disable nova-compute service on node (to take it out of scheduling)
>  3b) evacuate all existing VMs off of node

You mean s/evacuate/cold migrate/ of course... :)

>  3c) upgrade compute node (on restart, the compute node will see no
>      VMs running on the node and will construct the provider tree inside
>      update_provider_tree() with an appropriate set of child providers
>      and inventories on those child providers)
>  3d) enable nova-compute service on node
>
> Which is virtually identical to the "normal" upgrade process whenever
> there are significant changes to the compute node -- such as upgrading
> libvirt or the kernel.

Not necessarily. It's totally legit (and I expect quite common) to just
reboot the host to take kernel changes, bringing back all the instances
that were there when it resumes. The "normal" case of moving things
around slide-puzzle-style applies to live migration (which isn't an
option here). I think people that can take downtime for the instances
would rather not have to move things around for no reason if the
instance has to get shut off anyway.

> Nested resource tracking is another such significant change and should
> be dealt with in a similar way, IMHO.

This basically says that for anyone to move to rocky, they will have to
cold migrate every single instance in order to do that upgrade right? I
mean, anyone with two socket machines or SRIOV NICs would end up with at
least one level of nesting, correct? Forcing everyone to move everything
to do an upgrade seems like a non-starter to me.

We also need to consider the case where people would be FFU'ing past
rocky (i.e. never running rocky computes). We've previously said that
we'd provide a way to push any needed transitions with everything
offline to facilitate that case, so I think we need to implement that
method anyway.

I kinda think we need to either:

1. Make everything perform the pivot on compute node start (which can be
   re-used by a CLI tool for the offline case)
2. Make everything default to non-nested inventory at first, and provide
   a way to migrate a compute node and its instances one at a time (in
   place) to roll through.

We can also document "or do the cold-migration slide puzzle thing" as an
alternative for people that feel that's more reasonable.

I just think that forcing people to take down their data plane to work
around our own data model is kinda evil and something we should be
avoiding at this level of project maturity. What we're really saying is
"we know how to translate A into B, but we require you to move many GBs
of data over the network and take some downtime because it's easier for
*us* than making it seamless."

--Dan