Open Stack

On Aug 15, 2013, at 6:20 PM, Angus Salkeld <asalkeld at redhat.com> wrote:

> On 15/08/13 17:50 -0500, Christopher Armstrong wrote:
>> *Introduction and Requirements*
>> 
>> So there's kind of a perfect storm happening around autoscaling in Heat
>> right now. It's making it really hard to figure out how I should compose
>> this email. There are a lot of different requirements, a lot of different
>> cool ideas, and a lot of projects that want to take advantage of
>> autoscaling in one way or another: Trove, OpenShift, TripleO, just to name
>> a few...
>> 
>> I'll try to list the requirements from various people/projects that may be
>> relevant to autoscaling or scaling in general.
>> 
>> 1. Some users want a service like Amazon's Auto Scaling or Rackspace's
>> Otter -- a simple API that doesn't really involve orchestration.
>> 2. If such a API exists, it makes sense for Heat to take advantage of its
>> functionality instead of reimplementing it.
>> 3. If Heat integrates with that separate API, however, that API will need
>> two ways to do its work:
>>  1. native instance-launching functionality, for the "simple" use
>>  2. a way to talk back to Heat to perform orchestration-aware scaling
>> operations.
>> 4. There may be things that are different than AWS::EC2::Instance that we
>> would want to scale (I have personally been playing around with the concept
>> of a ResourceGroup, which would maintain a nested stack of resources based
>> on an arbitrary template snippet).
>> 5. Some people would like to be able to perform manual operations on an
>> instance group -- such as Clint Byrum's recent example of "remove instance
>> 4 from resource group A".
>> 
>> Please chime in with your additional requirements if you have any! Trove
>> and TripleO people, I'm looking at you :-)
>> 
>> 
>> *TL;DR*
>> 
>> Point 3.2. above is the main point of this email: exactly how should the
>> autoscaling API talk back to Heat to tell it to add more instances? I
>> included the other points so that we keep them in mind while considering a
>> solution.
>> 
>> *Possible Solutions*
>> 
>> I have heard at least three possibilities so far:
>> 
>> 1. the autoscaling API should maintain a full template of all the nodes in
>> the autoscaled nested stack, manipulate it locally when it wants to add or
>> remove instances, and post an update-stack to the nested-stack associated
>> with the InstanceGroup.
>> 
>> Pros: It doesn't require any changes to Heat.
>> 
>> Cons: It puts a lot of burden of state management on the autoscale API, and
>> it arguably spreads out the responsibility of "orchestration" to the
>> autoscale API. Also arguable is that automated agents outside of Heat
>> shouldn't be managing an "internal" template, which are typically developed
>> by devops people and kept in version control.
>> 
>> 2. There should be a new custom-built API for doing exactly what the
>> autoscaling service needs on an InstanceGroup, named something unashamedly
>> specific -- like "instance-group-adjust".
>> 
>> Pros: It'll do exactly what it needs to do for this use case; very little
>> state management in autoscale API; it lets Heat do all the orchestration
>> and only give very specific delegation to the external autoscale API.
>> 
>> Cons: The API grows an additional method for a specific use case.
> 
> I like this one above:
> adjust(new_size, victim_list=['i1','i7'])
> 
> So if you are reducing the new_size we look in the victim_list to
> choose those first. This should cover Clint's use case as well.
> 
> -Angus

We could just support victim_list=[1, 7], since these groups are collections of identical
resources. Simple indexing should be sufficient, I would think.

Perhaps separating the stimulus from the actions to take would let us design/build toward different policy implementations. Initially, we could have a HeatScalingPolicy that works with the signals that a scaling group can handle. When/if AS becomes an API outside of Heat, we can implement a fairly simple NovaScalingPolicy that includes the args to pass to nova boot.

>> 3. the autoscaling API should update the "Size" Property of the
>> InstanceGroup resource in the stack that it is placed in. This would
>> require the ability to PATCH a specific piece of a template (an operation
>> isomorphic to update-stack).

I think a PATCH semantic for updates would be generally useful in terms of "quality of life" for API users. Not having to pass the complete state and param values for trivial updates would be quite nice regardless of its implications to AS.

>> 
>> Pros: The API modification is generic, simply a more optimized version of
>> update-stack; very little state management required in autoscale API.
>> 
>> Cons: This would essentially require manipulating the user-provided
>> template.  (unless we have a concept of "private properties", which perhaps
>> wouldn't appear in the template as provided by the user, but can be
>> manipulated with such an update stack operation?)
>> 
>> 
>> *Addenda*
>> 
>> Keep in mind that there are use cases which require other types of
>> manipulation of the InstanceGroup -- not just the autoscaling API. For
>> example, see Clint's #5 above.
>> 
>> 
>> Also, about implementation: Andrew Plunk and I have begun work on Heat
>> resources for Rackspace's Otter, which I think will be a really good proof
>> of concept for how this stuff should work in the Heat-native autoscale API.
>> I am trying to gradually work the design into the native Heat autoscaling
>> design, and we will need to solve the autoscale-controlling-InstanceGroup
>> issue soon.
>> 
>> -- 
>> IRC: radix
>> Christopher Armstrong
>> Rackspace
> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev