Open Stack

On 4 April 2013 18:37, Adrian Otto <adrian.otto at rackspace.com> wrote:
> Alan,
>
> We want a useful Auto-Scale solution in Openstack, and recognize that Heat
> has some capability already that could be leveraged. Whether Auto-Scale is
> offered as a standalone service, by Heat, or (more likely) a combination of
> both is open for discussion. Duncan's team is focused exclusively on
> Auto-Scale, whereas our second team is concerned with the first three focus
> areas I mentioned. So I will look to Duncan to gather input from you on
> Auto-Scale.
>
> My take:
>
> Auto-scale is a control system. I believe that control systems should be
> simple, and well coordinated. We should be careful not to end up with
> multiple different control systems scattered about, especially in a complex
> system like ours where the controls may actually conflict with each other.
> Possible solutions include using centralized control, or perhaps well
> coordinated federated control. Central control is simpler, so I prefer that
> as a starting point. We could iterate toward something more elaborate as
> needed.

Agree with what you're saying here. I'd like to highlight one reason
why I'm cautious about the idea of splitting autoscale out of heat.

Consider rolling deployments with canaries: here you have (say) 1000
machines running Swift-proxy, and you need to deploy a new image to
them all. You start by deploying a small number (say 1), and check it
runs ok; then 10, and check, then 100 and check - note that at this
point 10% of your capacity is now deploying concurrently, or has
deployed but isn't yet clearly ok. If you have a load spike at this
point, it might be the *worst* possible thing would be to auto scale
up, because the load spike might be due to a bad image/config in the
new instances: adding more of the new image to the mix may just
increase the problem.

So autoscaling and deploys interact at a very deep level. What we need
to do to deal with that is an open question, but one things seems
clear - if Heat were to consume a separate auto scaling service, that
service would have to either not make Nova calls itself (and just
inform Heat as to recommended actions), or Heat would need to disable
it during deployments, which is going to be a huge chunk of the time
in busy environments.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Cloud Services