Open Stack

I respect your opinion but respectfully disagree that this is something 
we need to spend our time on. Comments inline.

On 08/29/2018 10:47 AM, Dan Smith wrote:
>>> * Cells can shard across flavors (and hardware type) so operators
>>> would like to move users off the old flavors/hardware (old cell) to
>>> new flavors in a new cell.
>>
>> So cell migrations are kind of the new release upgrade dance. Got it.
> 
> No, cell migrations are about moving instances between cells for
> whatever reason. If you have small cells for organization, then it's
> about not building arbitrary barriers between aisles. If you use it for
> hardware refresh, then it might be related to long term lifecycle. I'm
> not sure what any of this has to do with release upgrades or dancing.

A release upgrade dance involves coordination of multiple moving parts. 
It's about as similar to this scenario as I can imagine. And there's a 
reason release upgrades are not done entirely within Nova; clearly an 
external upgrade tool or script is needed to orchestrate the many steps 
and components involved in the upgrade process.

The similar dance for cross-cell migration is the coordination that 
needs to happen between Nova, Neutron and Cinder. It's called 
orchestration for a reason and is not what Nova is good at (as we've 
repeatedly seen)

  The thing that makes *this* particular scenario problematic is that 
cells aren't user-visible things. User-visible things could much more 
easily be orchestrated via external actors, as I still firmly believe 
this kind of thing should be done.

>> shelve was and continues to be a hack in order for users to keep an
>> IPv4 address while not consuming compute resources for some amount of
>> time. [1]
> 
> As we discussed in YVR most recently, it also may become an important
> thing for operators and users where expensive accelerators are committed
> to instances with part-time usage patterns.

I don't think that's a valid use case in respect to this scenario of 
cross-cell migration. If the target cell compute doesn't have the same 
expensive accelerators on them, nobody would want or permit a move to 
that target cell anyway.

Also, I'd love to hear from anyone in the real world who has 
successfully migrated (live or otherwise) an instance that "owns" 
expensive hardware (accelerators, SR-IOV PFs, GPUs or otherwise).

The patterns that I have seen are one of the following:

* Applications don't move. They are pets that stay on one or more VMs or 
baremetal nodes and they grow roots.

* Applications are designed to *utilize* the expensive hardware. They 
don't "own" the hardware itself.

In this latter case, the application is properly designed and stores its 
persistent data in a volume and doesn't keep state outside of the 
application volume. In these cases, the process of "migrating" an 
instance simply goes away. You just detach the application persistent 
volume, shut down the instance, start up a new one elsewhere (allowing 
the scheduler to select one that meets the resource constraints in the 
flavor/image), attach the volume again and off you go. No messing around 
with shelving, offloading, migrating, or any of that nonsense in Nova.

We should not pretend that what we're discussing here is anything other 
than hacking orchestration workarounds into Nova to handle 
poorly-designed applications that have grown roots on some hardware and 
think they "own" hardware resources in a Nova deployment.

> It has also come up more than once in the realm of "but I need to
> detach my root volume" scenarios. I love to hate on shelve as well,
> but recently a few more legit (than merely keeping an IPv4 address)
> use-cases have come out for it, and I don't think Matt is wrong that
> cross-cell migration *might* be easier as a shelve operation under
> the covers.

Matt may indeed be right, but I'm certainly allowed to express my 
opinion that I think shelve is a monstrosity that should be avoided at 
all costs and building additional orchestration functionality into Nova 
on top of an already-shaky foundation (shelve) isn't something I think 
is a long-term maintainable solution.

>> If cross-cell cold migration is similarly just about the user being
>> able to keep their instance's IPv4 address while allowing an admin to
>> move an instance's storage to another physical location, then my firm
>> belief is that this kind of activity needs to be coordinated
>> *externally to Nova*.
> 
> I'm not sure how you could make that jump, but no, I don't think that's
> the case. In any sort of large cloud that uses cells to solve problems
> of scale, I think it's quite likely to expect that your IPv4 address
> physically can't be honored in the target cell, and/or requires some
> less-than-ideal temporary tunneling for bridging the gap.

If that's the case, why are we discussing shelve at all? Just stop the 
instance, copy/migrate the volume data (if needed, again it completely 
depends on the deployment, network topology and block storage backend), 
to a new location (new cell, new AZ, new host agg, does it really 
matter?) and start a new instance, attaching the volume after the 
instance starts or supplying the volume in the boot/create command.

>> Since we're not talking about live migration (thank all that is holy),
> 
> Oh it's coming. Don't think it's not.
> 
>> I believe the safest and most effective way to perform such a
>> cross-cell "migration" would be the following basic steps:
>>
>> 0. ensure that each compute node is associated with at least one nova
>> host aggregate that is *only* in a single cell
>> 1. shut down the instance (optionally snapshotting required local disk
>> changes if the user is unfortunately using their root disk for
>> application data)
>> 2. "save" the instance's IP address by manually creating a port in
>> Neutron and assigning the IP address manually to that port. this of
>> course will be deployment-dependent since you will need to hope the
>> saved IP address for the migrating instance is in a subnet range that
>> is available in the target cell
>> 3. migrate the volume manually. this will be entirely deployment and
>> backend-dependent as smcginnis alluded to in a response to this thread
>> 4. have the admin boot the instance in a host aggregate that is known
>> to be in the target cell, passing --network
>> port_id=$SAVED_PORT_WITH_IP and --volume $MIGRATED_VOLUME_UUID
>> arguments as needed. the admin would need to do this because users
>> don't know about host aggregates and, frankly, the user shouldn't know
>> about host aggregates, cells, or any of this.
> 
> What you just described here is largely shelve, ignoring the volume
> migration part and the fact that such a manual process means the user
> loses the instance's uuid and various other elements about it (such as
> create time, action/event history, etc). Oh, and ignoring the fact that
> the user no longer owns their instance (the admin does) :)

The admin only "owns" the instance because we have no ability to 
transfer ownership of the instance and a cell isn't a user-visible 
thing. An external script that accomplishes this kind of orchestrated 
move from one cell to another could easily update the ownership of said 
instance in the DB.

My point is that Nova isn't an orchestrator, and building functionality 
into Nova to do this type of cross-cell migration IMHO just will lead to 
even more unmaintainable code paths that few, if any, deployers will 
ever end up using because they will end up doing it externally anyway 
due to the need to integrate with backend inventory management systems 
and other things.

Best,
-jay

> Especially given that migrating across a cell may mean "one aisle over,
> same storage provider and network" to a lot of people, the above being a
> completely manual process seems a little crazy to me.
> 
> --Dan
>