Open Stack

Excerpts from Zane Bitter's message of 2013-09-27 06:58:40 -0700:
> On 27/09/13 08:58, Mike Spreitzer wrote:
> > I have begun to draft some specifics about the sorts of policies that
> > might be added to infrastructure to inform a smart unified placement
> > engine.  These are cast as an extension to Heat templates.  See
> > https://wiki.openstack.org/wiki/Heat/PolicyExtension.  Comments solicited.
> 
> Mike,
> These are not the kinds of specifics that are of any help at all in 
> figuring out how (or, indeed, whether) to incorporate holistic 
> scheduling into OpenStack.

I agree that the things in that page are a wet dream of logical deployment
fun. However, I think one can target just a few of the basic ones,
and see a real achievable case forming. I think I grasp Mike's ideas,
so I'll respond to your concerns with what I think. Note that it is
highly likely I've gotten some of this wrong.

> 
> - What would a holistic scheduling service look like? A standalone 
> service? Part of heat-engine?

I see it as a preprocessor of sorts for the current infrastructure engine.
It would take the logical expression of the cluster and either turn
it into actual deployment instructions or respond to the user that it
cannot succeed. Ideally it would just extend the same Heat API.

> - How will the scheduling service reserve slots for resources in advance 
> of them being created? How will those reservations be accounted for and 
> billed?
> - In the event that slots are reserved but those reservations are not 
> taken up, what will happen?

I dont' see the word "reserve" in Mike's proposal, and I don't think this
is necessary for the more basic models like Collocation and Anti-Collocation.

Reservations would of course make the scheduling decisions more likely to
succeed, but it isn't necessary if we do things optimistically. If the
stack create or update fails, we can retry with better parameters.

> - Once scheduled, how will resources be created in their proper slots as 
> part of a Heat template?

In goes a Heat template (sorry for not using HOT.. still learning it. ;)

Resources:
  ServerTemplate:
    Type: Some::Defined::ProviderType
  HAThing1:
    Type: OS::Heat::HACluster
    Properties:
      ClusterSize: 3
      MaxPerAZ: 1
      PlacementStrategy: anti-collocation
      Resources: [ ServerTemplate ]

And if we have at least 2 AZ's available, it feeds to the heat engine:

Resources:
  HAThing1-0:
    Type: Some::Defined::ProviderType
      Parameters:
        availability-zone: zone-A
  HAThing1-1:
    Type: Some::Defined::ProviderType
      Parameters:
        availability-zone: zone-B
  HAThing1-2:
    Type: Some::Defined::ProviderType
      Parameters:
        availability-zone: zone-A

If not, holistic scheduler says back "I don't have enough AZ's to
satisfy MaxPerAZ".

Now, if Nova grows anti-affininty under the covers that it can manage
directly, a later version can just spit out:

Resources:
  HAThing1-0:
    Type: Some::Defined::ProviderType
      Parameters:
        instance-group: 0
        affinity-type: anti
  HAThing1-1:
    Type: Some::Defined::ProviderType
      Parameters:
        instance-group: 1
        affinity-type: anti
  HAThing1-2:
    Type: Some::Defined::ProviderType
      Parameters:
        instance-group: 0
        affinity-type: anti

The point is that the user cares about their servers not being in the
same failure domain, not how that happens.

> - What about when the user calls the APIs directly? (i.e. does their own 
> orchestration - either hand-rolled or using their own standalone Heat.)

This has come up with autoscaling too. "Undefined" - that's not your stack.

> - How and from where will the scheduling service obtain the utilisation 
> data needed to perform the scheduling? What mechanism will segregate 
> this information from the end user?

I do think this is a big missing piece. Right now it is spread out
all over the place. Keystone at least has regions, so that could be
incorporated now. I briefly dug through the other API's and don't see
a way to enumerate AZ's or cells. Perhaps it is hiding in extensions?

I don't think this must be segregated from end users. An API for "show
me the placement decisions I can make" seems useful for anybody trying
to automate deployments. Anyway, probably best to keep it decentralized
and just make it so that each service can respond with lists of arguments
to their API that are likely to succeed.

> - How will users communicate their scheduling constraints to OpenStack? 
> (Through which API and in what form?)

See above. Via the Heat API, a Heat-ish template that is turned into
concrete Heat instructions.

> - What value does this provide (over and above non-holistic scheduler 
> hints passed in individual API calls) to end users? Public cloud 
> operators? Private cloud operators? How might the value be shared 
> between users and operators, and how would that be accounted for?

See above, logically expressing what you actually want means the tool
can improve its response to that. Always expressing things concretely
means that improvements on the backend are harder to realize.

Ultimately, it is an end-user tool, but the benefit to a cloud operator
could be significant.  If one AZ is getting flooded, one can stop
responding to it, or add hints in the API ranking the AZ lower than
the others in preference. Users using the holistic scheduler will begin
using the new AZ without having to be educated about it.

> - Does this fit within the scope of an existing OpenStack program? Which 
> one? Why?

Heat. You want users to use holistic scheduling when it can work for them,
so having it just be a tweak to their templates is a win.

> - What changes are required to existing services to accommodate this 
> functionality?
> 

More exposure of what details can be exploited.