Open Stack

From: Vishvananda Ishaya [mailto:vishvananda at gmail.com]
Sent: Wednesday, February 11, 2015 5:07 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [cinder] Fixing stuck volumes - part II


On Feb 11, 2015, at 3:45 PM, D'Angelo, Scott <scott.dangelo at hp.com<mailto:scott.dangelo at hp.com>> wrote:


At the cinder mid-cycle it was decided that the best way to fix volumes stuck in 'attaching' or 'detaching' was NOT to fix the broken reset-state command. The doc string and help message for reset-state have been modified to warn the user that the tool only affects Cinder DB and can cause problems. But, ultimately, a separate command to 'force-detach' would be better. I've abandoned the original BP/spec for reset-state involving the driver.

I have looked at the existing function 'force-detach' in Cinder and it seems to work...except that Nova must be involved. Nova uses the BlockDeviceMapping table to keep track of attached volumes and, if Nova is not involved, a force-detach'ed volume will not be capable of being re-attached.
So, my plan is to submit a blueprint + spec for Novaclient to add a 'force-detach' command. This is technically fairly simple and only involves stripping away the checks for proper state in Nova, and calling Cinder force-detach. I don't plan on asking for an exception to feature freeze, unless there is optimism from the community that this could possible get in for L.
The existing Cinder force-detach calls terminate_connection() and detach_volume().  I assume detach_volume() is covered by the "Volume Detach" minimum feature? I see many drivers have terminate_connection(), but not all. I believe this will not be a minimum feature, but others may disagree.

If you are going to add a force-detach command to nova, I think it would be good to make it detach even if the cinder request fails. Currently if you try to detach a volume (or terminate an instance with an attached volume), if cinder is down or the volume node where the volume resides is down, nova refuses to continue, which is pretty bad user experience.

Vish

The only problem with that is, what happens when cinder comes back up? You have an user/admin who think that the volume should be available, but cinder DB still lists as attaching | detaching and the backend may still be exporting the volume to the Nova compute host.
We could expose 'force-detach' through the cinderclient to fix this, but the admin/user might think that this is the first place to start, and leave Nova out, which results in a volume that cannot be re-attached.
I think that you are right about user experience, but I'm cautious about other problems that might result.

thanks,
scottda
scott.dangelo at hp.com<mailto:scott.dangelo at hp.com>


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150212/6a70e872/attachment.html>