<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 9, 2016 at 3:23 PM, Ildikó Váncsa <span dir="ltr"><<a href="mailto:ildiko.vancsa@ericsson.com" target="_blank">ildiko.vancsa@ericsson.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Walt,<br>
<span class=""><br>
> -----Original Message-----<br>
> From: Walter A. Boring IV [mailto:<a href="mailto:walter.boring@hpe.com">walter.boring@hpe.com</a>]<br>
</span><span class="">> Sent: February 09, 2016 23:15<br>
> To: <a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><br>
> Subject: Re: [openstack-dev] [Nova][Cinder] Multi-attach, determining when to call os-brick's connector.disconnect_volume<br>
><br>
> On 02/09/2016 02:04 PM, Ildikó Váncsa wrote:<br>
> > Hi Walt,<br>
> ><br>
> > Thanks for starting this thread. It is a good summary of the issue and the proposal also looks feasible to me.<br>
> ><br>
> > I have a quick, hopefully not too wild idea based on the earlier discussions we had. We were considering earlier to store the target<br>
> identifier together with the other items of the attachment info. The problem with this idea is that when we call initialize_connection<br>
> from Nova, Cinder does not get the relevant information, like instance_id, to be able to do this. This means we cannot do that using<br>
> the functionality we have today.<br>
> ><br>
> > My idea here is to extend the Cinder API so that Nova can send the missing information after a successful attach. Nova should have<br>
> all the information including the 'target', which means that it could update the attachment information through the new Cinder API.<br>
> I think we need to do is to allow the connector to be passed at<br>
> os-attach time. Then cinder can save it in the attachment's table entry.<br>
><br>
> We will also need a new cinder API to allow that attachment to be updated during live migration, or the connector for the attachment<br>
> will get stale and incorrect.<br>
<br>
</span>When saying below that it will be good for live migration as well I meant that the update is part of the API.<br>
<br>
Ildikó<br>
<div><div class="h5"><br>
><br>
> Walt<br>
> ><br>
> > It would mean that when we request for the volume info from Cinder at detach time the 'attachments' list would contain all the<br>
> required information for each attachments the volume has. If we don't have the 'target' information because of any reason we can<br>
> still use the approach described below as fallback. This approach could even be used in case of live migration I think.<br>
> ><br>
> > The Cinder API extension would need to be added with a new microversion to avoid problems with older Cinder versions talking to<br>
> new Nova.<br>
> ><br>
> > The advantage of this direction is that we can reduce the round trips to Cinder at detach time. The round trip after a successful<br>
> attach should not have an impact on the normal operation as if that fails the only issue we have is we need to use the fall back method<br>
> to be able to detach properly. This would still affect only multiattached volumes, where we have more than one attachments on the<br>
> same host. By having the information stored in Cinder as well we can also avoid removing a target when there are still active<br>
> attachments connected to it.<br>
> ><br>
> > What do you think?<br>
> ><br>
> > Thanks,<br>
> > Ildikó<br>
> ><br>
> ><br>
> >> -----Original Message-----<br>
> >> From: Walter A. Boring IV [mailto:<a href="mailto:walter.boring@hpe.com">walter.boring@hpe.com</a>]<br>
> >> Sent: February 09, 2016 20:50<br>
> >> To: OpenStack Development Mailing List (not for usage questions)<br>
> >> Subject: [openstack-dev] [Nova][Cinder] Multi-attach, determining<br>
> >> when to call os-brick's connector.disconnect_volume<br>
> >><br>
> >> Hey folks,<br>
> >> One of the challenges we have faced with the ability to attach a<br>
> >> single volume to multiple instances, is how to correctly detach that<br>
> >> volume. The issue is a bit complex, but I'll try and explain the problem, and then describe one approach to solving one part of the<br>
> detach puzzle.<br>
> >><br>
> >> Problem:<br>
> >> When a volume is attached to multiple instances on the same host.<br>
> >> There are 2 scenarios here.<br>
> >><br>
> >> 1) Some Cinder drivers export a new target for every attachment<br>
> >> on a compute host. This means that you will get a new unique volume path on a host, which is then handed off to the VM<br>
> instance.<br>
> >><br>
> >> 2) Other Cinder drivers export a single target for all instances<br>
> >> on a compute host. This means that every instance on a single host, will reuse the same host volume path.<br>
> >><br>
> >><br>
> >> When a user issues a request to detach a volume, the workflow boils<br>
> >> down to first calling os-brick's connector.disconnect_volume before<br>
> >> calling Cinder's terminate_connection and detach. disconnect_volume's job is to remove the local volume from the host OS and<br>
> close any sessions.<br>
> >><br>
> >> There is no problem under scenario 1. Each disconnect_volume only<br>
> >> affects the attached volume in question and doesn't affect any other<br>
> >> VM using that same volume, because they are using a different path that has shown up on the host. It's a different target<br>
> exported from the Cinder backend/array.<br>
> >><br>
> >> The problem comes under scenario 2, where that single volume is<br>
> >> shared for every instance on the same compute host. Nova needs to be<br>
> >> careful and not call disconnect_volume if it's a shared volume, otherwise the first disconnect_volume call will nuke every instance's<br>
> access to that volume.<br>
> >><br>
> >><br>
> >> Proposed solution:<br>
> >> Nova needs to determine if the volume that's being detached is a shared or non shared volume. Here is one way to determine<br>
> that.<br>
> >><br>
> >> Every Cinder volume has a list of it's attachments. In those attachments it contains the instance_uuid that the volume is attached<br>
> to.<br>
> >> I presume Nova can find which of the volume attachments are on the<br>
> >> same host. Then Nova can call Cinder's initialize_connection for each of those attachments to get the target's connection_info<br>
> dictionary.<br>
> >> This connection_info dictionary describes how to connect to the<br>
> >> target on the cinder backend. If the target is shared, then each of<br>
> >> the connection_info dicts for each attachment on that host will be<br>
> >> identical. Then Nova would know that it's a shared target, and then<br>
> >> only call os-brick's disconnect_volume, if it's the last attachment on that host. I think at most 2 calls to cinder's initialize_connection<br>
> would suffice to determine if the volume is a shared target. This would only need to be done if the volume is multi-attach capable and<br>
> if there are more than 1 attachments on the same host, where the detach is happening.<br>
> >><br>
> >> Walt<br>
> >><br>
> >> _____________________________________________________________________<br>
</div></div>> >> _____ OpenStack Development Mailing List (not for usage questions)<br>
<span class="">> >> Unsubscribe:<br>
> >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > ______________________________________________________________________<br>
</span>> > ____ OpenStack Development Mailing List (not for usage questions)<br>
<span class="im">> > Unsubscribe:<br>
> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > .<br>
> ><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span><div class=""><div class="h5">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><div class="gmail_default" style="font-family:monospace,monospace">This may still be in fact the easiest way to handle this. The only other thing I am still somewhat torn on here is that maybe Nova should be doing ref-counting WRT shared connections and NOT send the detach in that scenario to begin with? </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">In the case of unique targets per-attach we already just "work", but if you are using the same target/attachment on a compute node for multiple instances, then you probably should keep track of that on the users end and not remove it while in use. That seems like the more "correct" way to deal with this, but maybe that's just me. Keep in mind we could also do the same ref-counting on the Cinder side if we so choose.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><div class="gmail_default">We talked about this at mid-cycle with the Nova team and I proposed independent targets for each connection on Cinder's side. We can still do that IMO but that doesn't seem to be a very popular idea. </div><div class="gmail_default"><br></div><div class="gmail_default">My point here is just that it seems like there might be a way to fix this without breaking compatibility in the API. Thoughts?</div></div><br></div></div>