Open Stack

tl;dr - do we need a REST API microversion for multi-attach support in nova?

The details:

The volume multi-attach series in nova, starting here [1], has run into 
an upgrade problem.

Basically, there is code in liberty which doesn't pass an instance uuid 
and volume id to the block_device_mappings table query to uniquely 
identify BDMs for operations like detach, swap volume and live 
migration. This is a problem when we are trying to introduce support for 
volume multiattach because today nova considers volume/instance as a 1:1 
relationship, but this introduces a 1:M relationship (oh, and the nova 
bdm table doesn't have any unique constraints enforcing data integrity - 
oops!).

So in Mitaka we are for sure going to land the code that makes sure that 
the operations which have a volume and instance get a BDM based on those 
keys. The only one that doesn't is assisted-snapshots, which is a REST 
API limitation, and we're going to block that case from multiattach 
support (when it lands). Dan Smith is starting that cleanup work here [2].

Ildiko has been asking how multiattach could be merged in nova in mitaka 
and actually supported in the REST API, DB API, compute and virt driver 
layers. Because of this query issue, and rolling upgrades, we could have 
a mitaka API talking to liberty computes. If we allow sending a 
multiattach volume to a liberty compute that already has attachments, 
that will fail.

We talked about adding a check in the API layer to see if there are any 
liberty computes running and if not, then we could allow the 2nd attach 
on a multi-attach volume, but that is racy and not fail-safe, which 
could mean a user could get a multi-attach request to pass in one case 
but fail in another (it's a problem related to how the service versions 
are cached in-memory per API worker and isn't something that can be 
resolved before mitaka release).

Ildiko was asking about adding a configuration option to the compute API 
which basically toggles the multi-attach feature. The thinking is that 
would default to False in mitaka and an operator would flip it to True 
once all the computes are upgraded to mitaka. We don't normally 
feature-toggle like this though, so it's not an attractive option really.

A lot of this also falls down in the aspect of how a user actually knows 
when the compute endpoint they are talking to supports multi-attach. If 
you're trying to attach a volume to a 2nd instance after it's already 
attached today, you'll get a 400 response.

If we did the config option idea and it was False, you'd get a 400 
response for that also, even if you were running mitaka-level compute 
API (but still had liberty computes).

Normally we advertise capability in the API via microversions. So I'm 
thinking what we need is basically a microversion in the volume attach 
REST API which says:

1. if you have a multiattach=True volume
2. that is already attached to instance 1
3. and you're trying to attach it to instance 2
4. that fails *unless* you're requesting a new enough microversion 
(opt-in to the feature).

That seems clear enough from the client perspective I think.

There is another wrinkle, however, which is not unique to this case.

There are only certain cinder backends (lvm) and nova virt drivers 
(libvirt) that will support multiattach. In the case of nova, you don't 
know if the multiattach will actually be accepted until we've cast off 
to the compute and find out if the virt driver supports it.

So even if we have a microversion in the REST API, and all of your 
computes are at >=mitaka, if you're hitting a non-libvirt compute node, 
it will fail (you get an instance fault in the case of volume attach, 
and a NoValidHost in the case of boot from volume).

I don't think we have a solution to that issue right now. I think John 
has been trying to sort that out somehow with the feature classification 
effort, but I'm unclear on the details.

But I think the first question is whether or not we require a 
microversion for multiattach support in the REST API purely as a signal 
to client code that a given cloud supports it, at least on certain nodes 
(running lvm + libvirt + mitaka).

[1] https://review.openstack.org/#/c/193133/
[2] https://review.openstack.org/#/c/267169/

-- 

Thanks,

Matt Riedemann