[openstack-dev] [nova][cinder] volumes stuck detaching attaching and force detach
Matt Riedemann
mriedem at linux.vnet.ibm.com
Wed Mar 2 18:34:08 UTC 2016
On 3/1/2016 11:36 PM, John Griffith wrote:
>
>
> On Tue, Mar 1, 2016 at 3:48 PM, Murray, Paul (HP Cloud) <pmurray at hpe.com
> <mailto:pmurray at hpe.com>> wrote:
>
>
> > -----Original Message-----
> > From: D'Angelo, Scott
> >
> > Matt, changing Nova to store the connector info at volume attach time does
> > help. Where the gap will remain is after Nova evacuation or live migration,
>
> This will happen with shelve as well I think. Volumes are not
> detached in shelve
> IIRC.
>
> > when that info will need to be updated in Cinder. We need to
> change the
> > Cinder API to have some mechanism to allow this.
> > We'd also like Cinder to store the appropriate info to allow a
> force-detach for
> > the cases where Nova cannot make the call to Cinder.
> > Ongoing work for this and related issues is tracked and discussed
> here:
> > https://etherpad.openstack.org/p/cinder-nova-api-changes
> >
> > Scott D'Angelo (scottda)
> > ________________________________________
> > From: Matt Riedemann [mriedem at linux.vnet.ibm.com
> <mailto:mriedem at linux.vnet.ibm.com>]
> > Sent: Monday, February 29, 2016 7:48 AM
> > To: openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>
> > Subject: Re: [openstack-dev] [nova][cinder] volumes stuck detaching
> > attaching and force detach
> >
> > On 2/22/2016 4:08 PM, Walter A. Boring IV wrote:
> > > On 02/22/2016 11:24 AM, John Garbutt wrote:
> > >> Hi,
> > >>
> > >> Just came up on IRC, when nova-compute gets killed half way
> through a
> > >> volume attach (i.e. no graceful shutdown), things get stuck in
> a bad
> > >> state, like volumes stuck in the attaching state.
> > >>
> > >> This looks like a new addition to this conversation:
> > >> http://lists.openstack.org/pipermail/openstack-dev/2015-
> > December/0826
> > >> 83.html
> > >>
> > >> And brings us back to this discussion:
> > >>
> https://blueprints.launchpad.net/nova/+spec/add-force-detach-to-nova
> > >>
> > >> What if we move our attention towards automatically recovering
> from
> > >> the above issue? I am wondering if we can look at making our
> usually
> > >> recovery code deal with the above situation:
> > >>
> > https://github.com/openstack/nova/blob/834b5a9e3a4f8c6ee2e3387845fc24
> > >> c79f4bf615/nova/compute/manager.py#L934
> > >>
> > >>
> > >> Did we get the Cinder APIs in place that enable the
> force-detach? I
> > >> think we did and it was this one?
> > >>
> https://blueprints.launchpad.net/python-cinderclient/+spec/nova-force
> > >> -detach-needs-cinderclient-api
> > >>
> > >>
> > >> I think diablo_rojo might be able to help dig for any bugs we have
> > >> related to this. I just wanted to get this idea out there before I
> > >> head out.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > __________________________________________________________
> > ___________
> > >> _____
> > >>
> > >> OpenStack Development Mailing List (not for usage questions)
> > >> Unsubscribe:
> > >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >> .
> > >>
> > > The problem is a little more complicated.
> > >
> > > In order for cinder backends to be able to do a force detach
> > > correctly, the Cinder driver needs to have the correct 'connector'
> > > dictionary passed in to terminate_connection. That connector
> > > dictionary is the collection of initiator side information
> which is gleaned
> > here:
> > >
> https://github.com/openstack/os-brick/blob/master/os_brick/initiator/c
> > > onnector.py#L99-L144
> > >
> > >
> > > The plan was to save that connector information in the Cinder
> > > volume_attachment table. When a force detach is called, Cinder has
> > > the existing connector saved if Nova doesn't have it. The
> problem was
> > > live migration. When you migrate to the destination n-cpu
> host, the
> > > connector that Cinder had is now out of date. There is no API in
> > > Cinder today to allow updating an existing attachment.
> > >
> > > So, the plan at the Mitaka summit was to add this new API, but it
> > > required microversions to land, which we still don't have in
> Cinder's
> > > API today.
> > >
> > >
> > > Walt
> > >
> > >
> > __________________________________________________________
> > ____________
> > > ____ OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > Regarding storing off the initial connector information from the attach, does
> > this [1] help bridge the gap? That adds the connector dict to the
> > connection_info dict that is serialized and stored in the nova
> > block_device_mappings table, and then in that patch is used to pass it to
> > terminate_connection in the case that the host has changed.
> >
> > [1]https://review.openstack.org/#/c/266095/
> >
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> > __________________________________________________________
> > ________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> >request at lists.openstack.org?subject:unsubscribe
> <http://request@lists.openstack.org?subject:unsubscribe>
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __________________________________________________________
> > ________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > request at lists.openstack.org?subject:unsubscribe
> <http://request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> I think Matt's patch is closer to where we need to be than some of the
> other paths that folks are on actually. There's a lot of complexity
> that's being proposed in my opinion that might not be needed. I could
> be wrong, but I've been looking at what got things sort of messy to
> begin with and it mostly revolves around diverging on what cinders
> initialize_connection and attach calls do.
>
> I worked on a couple of things that hopefully might help:
> [1] is a simple add to the devref in cinder just to help new driver
> writers get a better idea of what the expectations are here. I realize
> that there are a couple of drivers in Cinder that have already diverged
> here, but it's at least a guideline. I also realize that the workflow
> may be a bit different for devices like Ceph.
>
> The other thing that I started (haven't spent the time fixing up the
> horrid unit tests because I wanted feedback on the proposal first) is
> this patch here: [2]. The idea for the first step is to clean up the
> attach code that we have now and actually use some of the tables that
> Cinder provides.
>
> Particularly we have a volume_attachments table that could persist all
> of the connector info rather easily and more importantly return an
> attachment-id to the caller. We're already passing this info in, but
> sadly we're just not *doing* much with it.
>
> The idea here is that the caller (Nova or otherwise) doesn't need to
> mess with all this tracking of references, connectors etc. Instead, it
> just uses the returned attachment-id and that's it. Note that we
> ALREADY have all of the connector info etc during the initialize and
> then the additional info on the attach calls. We just weren't *doing*
> anything with it. Nova is done with a volume and wants to detach it...
> just call "terminate_connection" with the volume-id and the
> attachment-id it received during initialize_connection. Cinder
> can/should figure out what to do if it's a multi-attached volume on the
> same node or whatever.
>
> This makes detach a much simpler endeavor I think, because now you just
> provide the attachment-id, Cinder looks it up in it's DB and it has all
> of the info from the connector etc. I realize this would be a change in
> the API in the future, but adding the attachment ID is backward
> compatible and won't break any existing callers.
>
> So what about multi-attach? I mentioned that earlier and that would be
> the next piece of follow on work on the Cinder side if this idea is
> viewed as worth pursuing, but would require we agree to go this route on
> the Nova side as well. Currently there are some challenges with the
> current Cinder implementation of multi-attach but I think they're pretty
> easy to change around, and then we could get back to just using a unique
> attachment-id for every attach call and handle the business of sorting
> and tracking those things on the Cinder side. Again, trying to keep
> things simple and non-obtrusive for the consumer while maintaining
> backward compatibility. Honestly, I don't think that any consumer of
> Cinder (Nova) should have to know anything about all of these details.
> It should just simply be "Hey... Cinder; I'm attaching this volume *here*".
>
> This proposal covers the *special* drivers that can't reuse a Tgt IQN,
> or that work with things like Access-Groups. More importantly it
> provides all of the Cinder drivers more flexibility to do things in
> their own driver code if they need to while persisting all of the
> various data that might be desired.
>
> Anyway, I would really love to keep working on this if there's interest,
> but if folks feel the right direction is to build in shared logic
> between the two sides and pass connector objects with every call, that's
> fair enough. I do worry about complexity in some of what's being
> discussed though.
>
> [1]: https://review.openstack.org/#/c/285471/
> [2]: https://review.openstack.org/#/c/284867/
>
> Thanks,
> John
>
> PS
> I know, the devref doc isn't publishing, I've fixed that up and
> submitted a patch to get that addressed
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
I don't know the specifics of the cinder side changes but I like the
idea of cinder storing the connector information with the volume
attachment so that nova doesn't have to track that in the
block_device_mappings.connection_info serialized dict-o-doom.
Nova doesn't currently pass the attachment_id to the
terminate_connection API, but it does use it when calling os-detach, so
I don't really see a problem with nova passing the same attachment_id to
the terminate_connection API. If anything, nova would persist the
attachment_id in the block_device_mappings table which would make more
sense to me, although we can also look it up via volume ID and instance
uuid.
--
Thanks,
Matt Riedemann
More information about the OpenStack-dev
mailing list