Open Stack

On 04/10, Matt Riedemann wrote:
> On 10/4/2019 11:03 AM, Walter Boring wrote:
> >    I think if we don't have a host connector passed in and the
> > attachment record doesn't have a connector saved,
> > then that results in the volume manager not calling the cinder driver to
> > terminate_connection and return.
> > This also bypasses the driver's remove_export() which is the last chance
> > for a driver to unexport a volume.
>
> Two things:
>
> 1. Yeah if the existing legacy attachment record doesn't have a connector I
> was worried about not properly cleaning on for that old connection, which is
> something I mentioned before, but also as mentioned we potentially have that
> case when a server is deleted and we can't get to the compute host to get
> the host connector, right?
>

Hi,

Not really... In that case we still have the BDM info in the DB, so we
can just make the 3 Cinder REST API calls ourselves (begin_detaching,
terminate_connection and detach) to have the volume unmapped, the export
removed, and the volume return to available as usual, without needing to
go to the storage array manually.


> 2. If I were to use os-terminate_connection, I seem to have a tricky
> situation on the migration flow because right now I'm doing:
>
> a) create new attachment with host connector
> b) complete new attachment (put the volume back to in-use status)
>   - if this fails I attempt to delete the new attachment
> c) delete the legacy attachment - I intentionally left this until the end to
> make sure (a) and (b) were successful.
>
> If I change (c) to be os-terminate_connection, will that screw up the
> accounting on the attachment created in (a)?
>
> If I did the terminate_connection first (before creating a new attachment),
> could that leave a window of time where the volume is shown as not
> attached/in-use? Maybe not since it's not the begin_detaching/os-detach
> API...I'm fuzzy on the cinder volume state machine here.
>
> Or maybe the flow would become:
>
> a) create new attachment with host connector

This is a good idea in itself, but it's not taking into account weird
behaviors that some Cinder drivers may have when you call them twice to
initialize the connection on the same host.  Some drivers end up
creating a different mapping for the volume instead of returning the
existing one; we've had bugs like this before, and that's why Nova made
a change in its live instance migration code to not call
intialize_connection on the source host to get the connection_info for
detaching.


> b) terminate the connection for the legacy attachment
>   - if this fails, delete the new attachment created in (a)
> c) complete the new attachment created in (a)
>   - if this fails...?
>
> Without digging into the flow of a cold or live migration I want to say
> that's closer to what we do there, e.g. initialize_connection for the new
> host, terminate_connection for the old host, complete the new attachment.
>

I think any workaround we try to find has a good chance of resulting in
a good number of bugs.

In my opinion our options are:

1- Completely detach and re-attach the volume
2- Write new code in Cinder

The new code can be either a new action or we can just add a
microversion to attachment create to also accept "connection_info", and
when we provide connection_info on the call the method confirms that
it's a "migration" (the volume is 'in-use' and doesn't have any
attachments) and it doesn't bother to call the cinder-volume to export
and map the volume again and simply saves this information in the DB.

I know the solution it's not "clean/nice/elegant", and I'd rather go
with option 1, but that would be terrible user experience, so I'll
settle for a solution that doesn't add much code to Cinder, is simple
for Nova, and is less likely to result in bugs.

What do you think?

Regards,
Gorka.

PS: In this week's meeting we briefly discussed this topic and agreed to
continue the conversation here and retake it on next week's meeting.

> --
>
> Thanks,
>
> Matt
>