Open Stack

On Thursday morning the Nova and Cinder teams got together for a 
cross-project design summit session. The full etherpad is here [1].

This was all about volume multi-attach.

A subset of people from both teams were actually meeting weekly for four 
weeks leading up to the summit to hash out some details which we hoped 
to resolve at the summit. Unfortunately that didn't happen.

The first thing we wanted to settle was why do we actually want/need to 
support volume multi-attach? Because "Cinder added it to their API in 
Juno so Nova needs to support it" isn't a good reason. There are a few 
drivers for this feature:

* The need for active/active and active/hot standby scenarios which 
can't accept downtime due to attaching a new volume.

* Some database clusters, like Oracle RAC, require shared volumes. So 
Trove is a stakeholder for this feature also.

* Other legacy application use cases were brought up that essentially 
mean this is something they need to bridge a gap to adopting OpenStack.

So we agreed that while this is not really something we necessarily like 
(because of the non-cloud legacy application nature of it), it is 
necessary so we're going to continue trying to make it happen.

We then quickly went over what was completed in Mitaka and explained the 
detach issue we ran into. The problem is when you have more than one 
volume attached to the same instance on the same host, when you detach 
one of them, both of the volume connections actually get terminated on 
the host.

This problem is also complicated by the fact that some Cinder backends 
will create one attachment per export/volume, whereas others will 
multiplex all volumes onto one attachment.

Coming into the session we really had two competing solutions from the 
Cinder team, one from Walter Boring and one from John Griffith. However, 
during the session another idea was brought up from Dan Smith. The full 
details are in the etherpad, but it's really an idea to abstract the 
multiple volume attachments on the Cinder side that Nova only sees a 
single volume, so Nova wouldn't have to change any of it's API handling 
for Cinder volumes to be checking if they support multiattach or not, 
and thus have to have conditional logic spread all over Nova (API, 
compute, and virt drivers).

With Dan's idea we'd still have the disconnect/detach problem where Nova 
would need to check if it can disconnect from the host if there is only 
a single attachment left, but Nova has to do that regardless for all of 
the proposed solutions.

It sounded like John Griffith had a similar idea before when looking at 
this problem, and we spent a fair amount of time discussing it on both 
sides during the session.

At the end of the session, we (Nova) came away with the following next 
steps:

1. John Griffith (Cinder team) would work on a proof of concept for the 
abstracted volume idea.

2. Cinder would work on adding a volume migration test to Tempest to be 
run in the multi-node gate job (this needs to happen regardless). Scott 
D'Angelo is going to work on this.

3. Ildikó Váncsa was going to work on the Nova volume disconnect ref 
counting logic.

4. We'd meet on Friday during the Nova meetup session to discuss more 
details.

--

So what happened then was we met on Friday and found out that we were 
all speaking different languages on Thursday, because the Cinder team 
didn't think that they were going to be going with this new abstracted 
volume idea. After much wailing and gnashing of teeth we agreed to do a 
hangout shortly after the summit to try and get back on the same page. 
So we're doing that tomorrow (5/5) at 1600 UTC. This is going to be at 
least myself, Ildikó, Scott and Walter on the call. Walter has been 
creating diagrams of the flows through Nova and Cinder for various 
interactions like attach/detach of volumes and volume-backed live 
migration so that we can try to step back and see where the proposed 
changes fit in.

It's sounding like regardless of solution there will be changes to the 
Cinder API (at least for os-initialize_connection). There might be some 
new APIs too, for example, an API to get connection_info during live 
migration without Nova having to call os-initialize_connection to get it.

So we'll see what happens. We'll eventually figure out. After all, it's 
only code, right? :)

[1] https://etherpad.openstack.org/p/newton-nova-cinder

-- 

Thanks,

Matt Riedemann