Open Stack

On Wed, Jan 13, 2016 at 04:52:37PM -0600, Matt Riedemann wrote:
> tl;dr - do we need a REST API microversion for multi-attach support in nova?

Yes, I think so

> 
> The details:
> 
> The volume multi-attach series in nova, starting here [1], has run into an
> upgrade problem.
> 
> Basically, there is code in liberty which doesn't pass an instance uuid and
> volume id to the block_device_mappings table query to uniquely identify BDMs
> for operations like detach, swap volume and live migration. This is a
> problem when we are trying to introduce support for volume multiattach
> because today nova considers volume/instance as a 1:1 relationship, but this
> introduces a 1:M relationship (oh, and the nova bdm table doesn't have any
> unique constraints enforcing data integrity - oops!).
> 
> So in Mitaka we are for sure going to land the code that makes sure that the
> operations which have a volume and instance get a BDM based on those keys.
> The only one that doesn't is assisted-snapshots, which is a REST API
> limitation, and we're going to block that case from multiattach support
> (when it lands). Dan Smith is starting that cleanup work here [2].
> 
> Ildiko has been asking how multiattach could be merged in nova in mitaka and
> actually supported in the REST API, DB API, compute and virt driver layers.
> Because of this query issue, and rolling upgrades, we could have a mitaka
> API talking to liberty computes. If we allow sending a multiattach volume to
> a liberty compute that already has attachments, that will fail.
> 
> We talked about adding a check in the API layer to see if there are any
> liberty computes running and if not, then we could allow the 2nd attach on a
> multi-attach volume, but that is racy and not fail-safe, which could mean a
> user could get a multi-attach request to pass in one case but fail in
> another (it's a problem related to how the service versions are cached
> in-memory per API worker and isn't something that can be resolved before
> mitaka release).
> 
> Ildiko was asking about adding a configuration option to the compute API
> which basically toggles the multi-attach feature. The thinking is that would
> default to False in mitaka and an operator would flip it to True once all
> the computes are upgraded to mitaka. We don't normally feature-toggle like
> this though, so it's not an attractive option really.
> 
> A lot of this also falls down in the aspect of how a user actually knows
> when the compute endpoint they are talking to supports multi-attach. If
> you're trying to attach a volume to a 2nd instance after it's already
> attached today, you'll get a 400 response.
> 
> If we did the config option idea and it was False, you'd get a 400 response
> for that also, even if you were running mitaka-level compute API (but still
> had liberty computes).
> 
> Normally we advertise capability in the API via microversions. So I'm
> thinking what we need is basically a microversion in the volume attach REST
> API which says:
> 
> 1. if you have a multiattach=True volume
> 2. that is already attached to instance 1
> 3. and you're trying to attach it to instance 2
> 4. that fails *unless* you're requesting a new enough microversion (opt-in
> to the feature).
> 
> That seems clear enough from the client perspective I think.

While I don't think it's strictly required by the api change guidelines [3]
I think the API interactions and behavior here feel different enough to warrant
having a microversion. Ideally there should have been some versioning in the
cinder api around the multiattach support but that ship has already sailed.
Treating the volume attach case in nova as the traditional single attach case
by default and having to specify a new microversion to enable using multiple
attach will at least make it more explicit to users which I think is a good
thing.

> 
> There is another wrinkle, however, which is not unique to this case.
> 
> There are only certain cinder backends (lvm) and nova virt drivers (libvirt)
> that will support multiattach. In the case of nova, you don't know if the
> multiattach will actually be accepted until we've cast off to the compute
> and find out if the virt driver supports it.

I'm probably overlooking something major but shouldn't nova know if the virt
driver supports multiattach? If there are no computes with a compatible setup
why not just return an error and not even attempt the cast? I'm guessing all the
necessary info isn't in the DB which means there isn't a way to check this up
front.

> 
> So even if we have a microversion in the REST API, and all of your computes
> are at >=mitaka, if you're hitting a non-libvirt compute node, it will fail
> (you get an instance fault in the case of volume attach, and a NoValidHost
> in the case of boot from volume).
> 
> I don't think we have a solution to that issue right now. I think John has
> been trying to sort that out somehow with the feature classification effort,
> but I'm unclear on the details.
> 
> But I think the first question is whether or not we require a microversion
> for multiattach support in the REST API purely as a signal to client code
> that a given cloud supports it, at least on certain nodes (running lvm +
> libvirt + mitaka).

-Matt Treinish


> [1] https://review.openstack.org/#/c/193133/
> [2] https://review.openstack.org/#/c/267169/
[3] http://specs.openstack.org/openstack/api-wg/guidelines/evaluating_api_changes.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160113/9476e201/attachment.pgp>