Open Stack

On Tue, Mar 1, 2016 at 3:48 PM, Murray, Paul (HP Cloud) <pmurray at hpe.com>
wrote:

>
> > -----Original Message-----
> > From: D'Angelo, Scott
> >
> > Matt, changing Nova to store the connector info at volume attach time
> does
> > help. Where the gap will remain is after Nova evacuation or live
> migration,
>
> This will happen with shelve as well I think. Volumes are not detached in
> shelve
> IIRC.
>
> > when that info will need to be updated in Cinder. We need to change the
> > Cinder API to have some mechanism to allow this.
> > We'd also like Cinder to store the appropriate info to allow a
> force-detach for
> > the cases where Nova cannot make the call to Cinder.
> > Ongoing work for this and related issues is tracked and discussed here:
> > https://etherpad.openstack.org/p/cinder-nova-api-changes
> >
> > Scott D'Angelo (scottda)
> > ________________________________________
> > From: Matt Riedemann [mriedem at linux.vnet.ibm.com]
> > Sent: Monday, February 29, 2016 7:48 AM
> > To: openstack-dev at lists.openstack.org
> > Subject: Re: [openstack-dev] [nova][cinder] volumes stuck detaching
> > attaching and force detach
> >
> > On 2/22/2016 4:08 PM, Walter A. Boring IV wrote:
> > > On 02/22/2016 11:24 AM, John Garbutt wrote:
> > >> Hi,
> > >>
> > >> Just came up on IRC, when nova-compute gets killed half way through a
> > >> volume attach (i.e. no graceful shutdown), things get stuck in a bad
> > >> state, like volumes stuck in the attaching state.
> > >>
> > >> This looks like a new addition to this conversation:
> > >> http://lists.openstack.org/pipermail/openstack-dev/2015-
> > December/0826
> > >> 83.html
> > >>
> > >> And brings us back to this discussion:
> > >> https://blueprints.launchpad.net/nova/+spec/add-force-detach-to-nova
> > >>
> > >> What if we move our attention towards automatically recovering from
> > >> the above issue? I am wondering if we can look at making our usually
> > >> recovery code deal with the above situation:
> > >>
> > https://github.com/openstack/nova/blob/834b5a9e3a4f8c6ee2e3387845fc24
> > >> c79f4bf615/nova/compute/manager.py#L934
> > >>
> > >>
> > >> Did we get the Cinder APIs in place that enable the force-detach? I
> > >> think we did and it was this one?
> > >> https://blueprints.launchpad.net/python-cinderclient/+spec/nova-force
> > >> -detach-needs-cinderclient-api
> > >>
> > >>
> > >> I think diablo_rojo might be able to help dig for any bugs we have
> > >> related to this. I just wanted to get this idea out there before I
> > >> head out.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > __________________________________________________________
> > ___________
> > >> _____
> > >>
> > >> OpenStack Development Mailing List (not for usage questions)
> > >> Unsubscribe:
> > >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >> .
> > >>
> > > The problem is a little more complicated.
> > >
> > > In order for cinder backends to be able to do a force detach
> > > correctly, the Cinder driver needs to have the correct 'connector'
> > > dictionary passed in to terminate_connection.  That connector
> > > dictionary is the collection of initiator side information which is
> gleaned
> > here:
> > > https://github.com/openstack/os-brick/blob/master/os_brick/initiator/c
> > > onnector.py#L99-L144
> > >
> > >
> > > The plan was to save that connector information in the Cinder
> > > volume_attachment table.  When a force detach is called, Cinder has
> > > the existing connector saved if Nova doesn't have it.  The problem was
> > > live migration.  When you migrate to the destination n-cpu host, the
> > > connector that Cinder had is now out of date.  There is no API in
> > > Cinder today to allow updating an existing attachment.
> > >
> > > So, the plan at the Mitaka summit was to add this new API, but it
> > > required microversions to land, which we still don't have in Cinder's
> > > API today.
> > >
> > >
> > > Walt
> > >
> > >
> > __________________________________________________________
> > ____________
> > > ____ OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > Regarding storing off the initial connector information from the attach,
> does
> > this [1] help bridge the gap? That adds the connector dict to the
> > connection_info dict that is serialized and stored in the nova
> > block_device_mappings table, and then in that patch is used to pass it to
> > terminate_connection in the case that the host has changed.
> >
> > [1] https://review.openstack.org/#/c/266095/
> >
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> > __________________________________________________________
> > ________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __________________________________________________________
> > ________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

​I think Matt's patch is closer to where we need to be than some of the
other paths that folks are on actually.  There's a lot of complexity that's
being proposed in my opinion that might not be needed.  I could be wrong,
but I've been looking at what got things sort of messy to begin with and it
mostly revolves around diverging on what cinders initialize_connection and
attach calls do.

I worked on a couple of things that hopefully might help:
[1] is a simple add to the devref in cinder just to help new driver writers
get a better idea of what the expectations are here.  I realize that there
are a couple of drivers in Cinder that have already diverged here, but it's
at least a guideline.  I also realize that the workflow may be a bit
different for devices like Ceph.​

The other thing that I started (haven't spent the time fixing up the horrid
unit tests because I wanted feedback on the proposal first) is this patch
here: [2].  The idea for the first step is to clean up the attach code that
we have now and actually use some of the tables that Cinder provides.

Particularly we have a volume_attachments table that could persist all of
the connector info rather easily and more importantly return an
attachment-id to the caller.  We're already passing this info in, but sadly
we're just not *doing* much with it.

The idea here is that the caller (Nova or otherwise) doesn't need to mess
with all this tracking of references, connectors etc. Instead, it just uses
the returned attachment-id and that's it.  Note that we ALREADY have all of
the connector info etc during the initialize and then the additional info
on the attach calls.  We just weren't *doing* anything with it.  Nova is
done with a volume and wants to detach it... just call
"terminate_connection" with the volume-id and the attachment-id it received
during initialize_connection.  Cinder can/should figure out what to do if
it's a multi-attached volume on the same node or whatever.

This makes detach a much simpler endeavor I think, because now you just
provide the attachment-id, Cinder looks it up in it's DB and it has all of
the info from the connector etc.  I realize this would be a change in the
API in the future, but adding the attachment ID is backward compatible and
won't break any existing callers.

So what about multi-attach?  I mentioned that earlier and that would be the
next piece of follow on work on the Cinder side if this idea is viewed as
worth pursuing, but would require we agree to go this route on the Nova
side as well.  Currently there are some challenges with the current Cinder
implementation of multi-attach but I think they're pretty easy to change
around, and then we could get back to just using a unique attachment-id for
every attach call and handle the business of sorting and tracking those
things on the Cinder side.  Again, trying to keep things simple and
non-obtrusive for the consumer while maintaining backward compatibility.
Honestly, I don't think that any consumer of Cinder (Nova) should have to
know anything about all of these details.  It should just simply be "Hey...
Cinder; I'm attaching this volume *here*".

This proposal covers the *special* drivers that can't reuse a Tgt IQN, or
that work with things like Access-Groups.  More importantly it provides all
of the Cinder drivers more flexibility to do things in their own driver
code if they need to while persisting all of the various data that might be
desired.

Anyway, I would really love to keep working on this if there's interest,
but if folks feel the right direction is to build in shared logic between
the two sides and pass connector objects with every call, that's fair
enough.  I do worry about complexity in some of what's being discussed
though.

[1]: https://review.openstack.org/#/c/285471/
[2]: https://review.openstack.org/#/c/284867/

Thanks,
John

PS
I know, the devref doc isn't publishing, I've fixed that up and submitted a
patch to get that addressed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160301/47196bf2/attachment.html>