Open Stack

On 07/04/15 22:02, Miguel Grinberg wrote:
> Hi,
>
> The OS::Heat::AutoScalingGroup resource is somewhat limited at this
> time, because when a scaling even occurs it does not notify dependent
> resources, such as a load balancer, that the pool of instances has changed.

As Thomas mentioned, the 'approved' way to solve this is to make your 
scaled unit a stack, and include a Neutron PoolMember resource in it.

> The AWS::AutoScaling::AutoScalingGroup resource, on the other side, has
> a LoadBalancerNames property that takes a list of
> AWS::ElasticLoadBalancing::LoadBalancer resources that get updated
> anytime the size of the ASG changes.

Which is an appalling hack.

(If it called the Neutron LBaaS API, like the equivalent in 
CloudFormation does with ELB, it would be OK. But in reality, as you 
know, it's a hack that makes calls directly to another resource plugin 
within Heat.)

> I'm trying to implement this notification mechanism for HOT templates,
> but there are a few aspects that I hope to do better.
>
> 1. A HOT template can have get_attr function calls that invoke
> attributes of the ASG. None of these update when the ASG resizes at this
> time, a scaling even does a partial update that only affects the ASG
> resource. I would like to address this.

In the medium-term I think this is something that I believe Convergence 
will be able to solve for us. I'm not sure that it's worth putting in a 
short-term work-around for.

> 2. The AWS solution relies on the well known LoadBalancer resource, but
> often load balancers are just regular instances that get loaded with a
> load balancer such as haproxy in a custom way. I'd like custom load
> balancers to also update when the ASG resizes.

TBH the correct answer for load balancers specifically is use the 
Neutron LBaaS API, end of story. But you're right that there are many 
uses for a more generic notification mechanism. (For example, in 
OpenShift we need to notify the controller when we add or remove nodes.) 
The design goal for ASG was always that we would have an arbitrary 
scaled unit (defined by a template) and an arbitrary non-scaled unit 
that could receive notifications about changes to the scaling group. So 
far we have delivered on only the first part of that promise.

My vision for the second part has always been that we'd use hooks, the 
initial implementation of which Tomas has landed in Kilo. We'll need to 
implement some more hook types to do it - post-create, post-update and 
pre-delete at a minimum. We also need some way of notifying the user 
asynchronously about when the hooks are triggered, so that they can take 
whatever action (e.g. add to load balancer) before calling the API to 
clear the hook. (At the moment the only way to find out when your hook 
should run is by polling the Heat API.)

In my ideal world, the notification mechanism (or at least one of them) 
is a message to a Zaqar queue/topic (whatever you want to call it) 
specified by the user. So someone e.g. running their own HAProxy (don't 
do this ;) could put a little micro-daemon on the same box that listened 
to Zaqar for notifications and update the HAProxy config appropriately.

Also in my ideal world, a Mistral workflow could be triggered (and 
seeded with parameter values) by the exact same message queue, so that 
the user can run any action that Mistral can support without having to 
have a server around to run it. And we'd use the same system for e.g. 
Ceilometer alarms talking to scaling policies, so that one could also 
insert a Mistral workflow into the process. Things are actually pretty 
awesome in my ideal world.

> The ResourceGroup is an interesting resource. It is much simpler than
> the ASG. In particular, the only way to scale the ResourceGroup is by
> issuing a stack-update with a new size. This indirectly solves #1 and #2
> above, because when a full update is issued any references to the
> ResourceGroup get updated as well.

It doesn't really solve the problem, because you could still manually 
update the nested stack that the ResourceGroup manages. It just entirely 
lacks the feature that makes it easy to run in to the problem. And not 
in a good way.

> In my opinion, the best way to address #1 and #2 above so that they work
> for the ASG as they work for the RG, is to change what happens when
> there is a scaling event. When the ScalingPolicy resource gets a signal,
> it reaches directly to the ASG by calling asg.adjust() (or in the near
> future by sending a signal to it, when a currently proposed patch
> merges) with the new size. This bypasses the update mechanism, so only a
> partial update occurs, just the ASG resource itself is updated. I would
> like this to be a full stack update, so that all references get updated
> with the new ASG size. This will address #1 and #2.

-1

The way to think about autoscaling is as a separate service that 
delegates the creation and deletion of its members to and maintains its 
state in a Heat stack. It *isn't* of course, but nor will it ever be if 
people continue to think about it as a resource plugin that is free to 
reach in to its parent stack and start messing with other things.

Apart from being a layering violation, anything that relies on updating 
the parent stack *after* a scaling operation is complete simply doesn't 
work. When scaling down, you want the changes to be made *before* 
updating the scaling group. In the general case - a batched rolling 
update - there are multiple changes that need to be made mostly *during* 
the scaling group update.

> But there is an alternative to this. I guess we could copy the update
> mechanism used on the AWS side, which is also partial, but at least

-2! This is what we most wanted to avoid in the native resources.

> covers the load balancers, given in the LoadBalancerNames property. We
> can have a "load_balancer_names" equivalent property for the
> OS::Heat::ASG resource, and we can then trigger the updates of the load
> balancer(s) exactly like the AWS side does it. For this option, I would
> like to extend the load balancer update mechanism to work on custom load
> balancers, as it currently works with the well known load balancer
> resources. I have implemented this approach and is currently up for
> review: https://review.openstack.org/#/c/170634/. I honestly prefer the
> full update, seems cleaner to me.
>
> Anyway, sorry for the long email. If you can provide guidance on which
> of the approaches are preferred, or if you have other ideas, I would
> appreciate it.

Long emails are good, thanks for writing this up :)

cheers,
Zane.