Open Stack

On 5/15/2017 2:28 PM, Edmund Rhudy (BLOOMBERG/ 120 PARK) wrote:
> Hi all,
>
> I'd like to follow up on a few discussions that took place last week in
> Boston, specifically in the Compute Instance/Volume Affinity for HPC
> session
> (https://etherpad.openstack.org/p/BOS-forum-compute-instance-volume-affinity-hpc).
>
> In this session, the discussions all trended towards adding more
> complexity to the Nova UX, like adding --near and --distance flags to
> the nova boot command to have the scheduler figure out how to place an
> instance near some other resource, adding more fields to flavors or
> flavor extra specs, etc.
>
> My question is: is it the right question to ask how to add more
> fine-grained complications to the OpenStack user experience to support
> what seemed like a pretty narrow use case?

I think we can all agree we don't want to complicate the user experience.

>
> The only use case that I remember hearing was an operator not wanting it
> to be possible for a user to launch an instance in a particular Nova AZ
> and then not be able to attach a volume from a different Cinder AZ, or
> they try to boot an instance from a volume in the wrong place and get a
> failure to launch. This seems okay to me, though - either the user has
> to rebuild their instance in the right place or Nova will just return an
> error during instance build. Is it worth adding all sorts of
> convolutions to Nova to avoid the possibility that somebody might have
> to build instances a second time?

We might have gone down this path but it's not the intention or the use 
case as I thought I had presented it, and is in the etherpad. For what 
you're describing, we already have the CONF.cinder.cross_az_attach 
option in nova which prevents you from booting or attaching a volume to 
an instance in a different AZ from the instance. That's not what we're 
talking about though.

The use case, as I got from the mailing list discussion linked in the 
etherpad, is a user wants their volume attached as close to local 
storage for the instance as possible for performance reasons. If this 
could be on the same physical server, great. But there is the case where 
the operator doesn't want to use any local disk on the compute and wants 
to send everything to Cinder, and the backing storage might not be on 
the same physical server, so that's where we started talking about 
--near or --distance (host, rack, row, data center, etc).

>
> The feedback I get from my cloud-experienced users most frequently is
> that they want to know why the OpenStack user experience in the storage
> area is so radically different from AWS, which is what they all have
> experience with. I don't really have a great answer for them, except to
> admit that in our clouds they just have to know what combination of
> flavors and Horizon options or BDM structure is going to get them the
> right tradeoff between storage durability and speed. I was pleased with
> how the session on expanding Cinder's role for Nova ephemeral storage
> went because of the suggestion of reducing Nova imagebackend's role to
> just the file driver and having Cinder take over for everything else.
> That, to me, is the kind of simplification that's a win-win for both
> devs and ops: devs get to radically simplify a thorny part of the Nova
> codebase, storage driver development only has to happen in Cinder,
> operators get a storage workflow that's easier to explain to users.
>
> Am I off base in the view of not wanting to add more options to nova
> boot and more logic to the scheduler? I know the AWS comparison is a
> little North America-centric (this came up at the summit a few times
> that EMEA/APAC operators may have very different ideas of a normal cloud
> workflow), but I am striving to give my users a private cloud that I can
> define for them in terms of AWS workflows and vocabulary. AWS by design
> restricts where your volumes can live (you can use instance store
> volumes and that data is gone on reboot or terminate, or you can put EBS
> volumes in a particular AZ and mount them on instances in that AZ), and
> I don't think that's a bad thing, because it makes it easy for the users
> to understand the contract they're getting from the platform when it
> comes to where their data is stored and what instances they can attach
> it to.
>

Again, we don't want to make the UX more complicated, but as noted in 
the etherpad, the solution we have today is if you want the same 
instance and volume on the same host for performance reasons, then you 
need to have a 1:1 relationship for AZs and hosts since AZs are exposed 
to the user. In a public cloud where you've got hundreds of thousands of 
compute hosts, 1:1 AZs aren't going to be realistic, for neither the 
admin or user. Plus, AZs are really supposed to be about fault domains, 
not performance domains, as Jay Pipes pointed out in the session.

That's where the idea of a --near or --distance=0 came in. I agree that 
having non-standard definitions of 'distance' is going to be confusing 
and not interoperable, so that's a whole other ball of wax, but it does 
provide flexibility for the operator, i.e. in my cloud, 'near' means 
same rack, but in some other cloud 'near' means the same data center.

If there are alternative ideas on how to design or model this, I'm all 
ears. Or if some OpenStack clouds are already providing ways to do this, 
I'd love to hear how they are doing it (I asked in the session and there 
are some replies in the etherpad but we didn't get into details).

-- 

Thanks,

Matt