Open Stack

On 16/08/13 00:50, Christopher Armstrong wrote:
> *Introduction and Requirements*
>
> So there's kind of a perfect storm happening around autoscaling in Heat
> right now. It's making it really hard to figure out how I should compose
> this email. There are a lot of different requirements, a lot of
> different cool ideas, and a lot of projects that want to take advantage
> of autoscaling in one way or another: Trove, OpenShift, TripleO, just to
> name a few...
>
> I'll try to list the requirements from various people/projects that may
> be relevant to autoscaling or scaling in general.
>
> 1. Some users want a service like Amazon's Auto Scaling or Rackspace's
> Otter -- a simple API that doesn't really involve orchestration.
> 2. If such a API exists, it makes sense for Heat to take advantage of
> its functionality instead of reimplementing it.

+1, obviously. But the other half of the story is that the API is likely 
be implemented using Heat on the back end, amongst other reasons because 
that implementation already exists. (As you know, since you wrote it ;)

So, just as we will have an RDS resource in Heat that calls Trove, and 
Trove will use Heat for orchestration:

   user => [Heat =>] Trove => Heat => Nova

there will be a similar workflow for Autoscaling:

   user => [Heat =>] Autoscaling -> Heat => Nova

where the first, optional, Heat stack contains the RDS/Autoscaling 
resource and the backend Heat stack contains the actual Nova instance(s).

One difference might be that the Autoscaling -> Heat step need not 
happen via the public ReST API. Since both are part of the Heat project, 
I think it would also be OK to do this over RPC only.

> 3. If Heat integrates with that separate API, however, that API will
> need two ways to do its work:

Wut?

>     1. native instance-launching functionality, for the "simple" use

This is just the simplest possible case of 3.2. Why would we maintain a 
completely different implementation?

>     2. a way to talk back to Heat to perform orchestration-aware scaling
> operations.

[IRC discussions clarified this to mean scaling arbitrary resource 
types, rather than just Nova servers.]

> 4. There may be things that are different than AWS::EC2::Instance that
> we would want to scale (I have personally been playing around with the
> concept of a ResourceGroup, which would maintain a nested stack of
> resources based on an arbitrary template snippet).
> 5. Some people would like to be able to perform manual operations on an
> instance group -- such as Clint Byrum's recent example of "remove
> instance 4 from resource group A".
>
> Please chime in with your additional requirements if you have any! Trove
> and TripleO people, I'm looking at you :-)
>
>
> *TL;DR*
>
> Point 3.2. above is the main point of this email: exactly how should the
> autoscaling API talk back to Heat to tell it to add more instances? I
> included the other points so that we keep them in mind while considering
> a solution.
>
> *Possible Solutions*
>
> I have heard at least three possibilities so far:
>
> 1. the autoscaling API should maintain a full template of all the nodes
> in the autoscaled nested stack, manipulate it locally when it wants to
> add or remove instances, and post an update-stack to the nested-stack
> associated with the InstanceGroup.

This is what I had been thinking.

> Pros: It doesn't require any changes to Heat.
>
> Cons: It puts a lot of burden of state management on the autoscale API,

All other APIs need to manage state too, I don't really have a problem 
with that. It already has to handle e.g. the cooldown state; your 
scaling strategy (uh, for the service) will be determined by that.

> and it arguably spreads out the responsibility of "orchestration" to the
> autoscale API.

Another line of argument would be that this is not true by definition ;)

> Also arguable is that automated agents outside of Heat
> shouldn't be managing an "internal" template, which are typically
> developed by devops people and kept in version control.
>
> 2. There should be a new custom-built API for doing exactly what the
> autoscaling service needs on an InstanceGroup, named something
> unashamedly specific -- like "instance-group-adjust".

+1 to having a custom (RPC-only) API if it means forcing some state out 
of the autoscaling service.

-1 for it talking to an InstanceGroup - that just brings back all our 
old problems about having "resources" that don't have their own separate 
state and APIs, but just exist inside of Heat plugins. Those are the 
cause of all of the biggest design problems in Heat. They're the thing I 
want the Autoscaling API to get rid of. (Also, see below.)

> Pros: It'll do exactly what it needs to do for this use case; very
> little state management in autoscale API; it lets Heat do all the
> orchestration and only give very specific delegation to the external
> autoscale API.
>
> Cons: The API grows an additional method for a specific use case.
>
> 3. the autoscaling API should update the "Size" Property of the
> InstanceGroup resource in the stack that it is placed in. This would
> require the ability to PATCH a specific piece of a template (an
> operation isomorphic to update-stack).

-1

Half the point of a separate autoscaling API is that an autoscaling 
group shouldn't _need_ to exist only in the context of a template.

So, to get around this I think you're proposing something where the two 
ways of interacting with Autoscaling are (with and without Heat):

   user => Heat <=> Autoscaling
             \
              ------------------> Heat => Nova

   user => Autoscaling => Heat -> Heat => Nova

Instead of:

   user => Heat => Autoscaling => Heat => Nova

   user =========> Autoscaling => Heat => Nova


This is two lots of code to write & test and, from the user's point of 
view, a bizarre and unprecedented conceptual inversion of where 
Orchestration belongs in the system. (Note that the backend Heat is an 
implementation detail hidden from the user in either case.)

cheers,
Zane.

> Pros: The API modification is generic, simply a more optimized version
> of update-stack; very little state management required in autoscale API.
>
> Cons: This would essentially require manipulating the user-provided
> template.  (unless we have a concept of "private properties", which
> perhaps wouldn't appear in the template as provided by the user, but can
> be manipulated with such an update stack operation?)
>
>
> *Addenda*
>
> Keep in mind that there are use cases which require other types of
> manipulation of the InstanceGroup -- not just the autoscaling API. For
> example, see Clint's #5 above.
>
>
> Also, about implementation: Andrew Plunk and I have begun work on Heat
> resources for Rackspace's Otter, which I think will be a really good
> proof of concept for how this stuff should work in the Heat-native
> autoscale API. I am trying to gradually work the design into the native
> Heat autoscaling design, and we will need to solve the
> autoscale-controlling-InstanceGroup issue soon.
>
> --
> IRC: radix
> Christopher Armstrong
> Rackspace
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>