Open Stack

On Thursday at the summit the Cinder team had a session about the 
proposed volume attach/detach API changes that John Griffith is working 
on. The etherpad for that session is here:

https://etherpad.openstack.org/p/ocata-nova-summit-cinder-session

On Friday we had a joint nova/cinder session to go over the proposals 
and POCs with the larger teams. The etherpad for that session is here:

https://etherpad.openstack.org/p/ocata-nova-summit-cinder-session

John Griffith has been working on two different versions of adding a new 
series of APIs that deal with attachment CRUD operations which nova 
would plan on using, and which should also help ease in the support for 
volume multiattach.

John's first iteration on this is slightly closer to what we already 
have today, but with Cinder storing more information (like the host 
connector and connection_info) and providing more information to Nova 
like whether or not a connection is shared - which varies based on the 
Cinder backend. That version still relies on nova to reserve (lock) the 
volume in nova-api before casting to nova-compute to create the 
attachment and connect it on the host using os-brick.

John is working on a second POC here:

https://review.openstack.org/#/c/387712/

That is slightly different and might eliminate the need for nova to call 
both os-reserve and create_attachment. Ideally nova gets to the point of 
simply creating the attachment (which locks the volume) in nova-api, 
then in nova-compute nova will update the attachment with the host 
connector (from os-brick) and then connect the volume on the host (using 
os-brick). If anything fails, nova would delete the attachment record in 
Cinder, thus unlocking the volume (like detach, terminate-connection and 
unreserve today).

One of the main issues we're still trying to sort out is nova knowing 
when it's safe to disconnect an attachment, mainly in the case of shared 
storage backends. We could have a race where nova is getting a 
connection count from cinder which basically says it's safe to 
disconnect from the host because it's the only connection (A), but 
meanwhile another request creates an attachment (B) and when nova 
disconnects A is also disconnects B. We think that we can work through 
this with a lock in nova so that you can't be attaching with a shared 
connection on a host at the same time that the shared connection is 
being detached from that host.

We also need to sort out the live migration flow where you have a 
non-multiattach volume but you're using it to create two connections for 
the same instance but on different hosts. There would most likely need 
to be some logic in the Cinder API to allow this, since otherwise 
normally you can't have more than one attachment to a non-multiattach 
volume. Supporting this, though, would make the volume-backed live 
migration flow in Nova quite simple, and would be similar to how 
Nova/Neutron plan to handle multiple port bindings during live migration.

With respect to upgrades, we haven't gotten too in-depth into that yet. 
With the proposed flows Nova would be tracking an attachment_id in the 
block_device_mappings for an instance. Nova may or may not be storing 
the connection_info anymore since Cinder would store that and Nova can 
just get it via the attachment record. But as part of an upgrade we'd 
likely need to migrate old existing BDM records to create an attachment 
record in Cinder and then store that in the nova BDM. It's TBD on if 
something like that could happen with a nova-manage command, or if it 
would be more of an online migration when operations are performed on 
older volumes.

The nova side flows with all of this are detailed in John Garbutt's nova 
spec here:

https://review.openstack.org/#/c/373203/

We're still having weekly nova/cinder cross-project meetings at 1700 UTC 
on Mondays:

http://eavesdrop.openstack.org/#Cinder,_Nova

And have started doing those with Google Hangouts to speed along the 
discussion. Ildikov has been taking notes during the IRC meeting though 
so the minutes/highlights are tracked in case the hangout isn't recorded.

All in all we are making progress, slowly but surely, on the design and 
working through some of the edge cases. If we can come to an agreement 
on the overall workflow early enough in Ocata then it might be possible 
to get the Cinder API changes in and then the Nova side can possibly get 
landed in Pike. However, given the Cinder team's focus on technical debt 
and stability for Ocata it might be a stretch. Either way, having an 
agreed on design and POC code by the end of Ocata would be a major 
accomplishment and greatly help getting volume multiattach supported in 
Pike.

-- 

Thanks,

Matt Riedemann