Open Stack

On Fri, Nov 28, 2014 at 02:33:10PM +0000, Jastrzebski, Michal wrote:
> 
> 
> > -----Original Message-----
> > From: Qiming Teng [mailto:tengqim at linux.vnet.ibm.com]
> > Sent: Friday, November 28, 2014 8:33 AM
> > To: openstack-dev at lists.openstack.org
> > Subject: [openstack-dev] [Heat] Rework auto-scaling support in Heat
> > 
> > Dear all,
> > 
> > Auto-Scaling is an important feature supported by Heat and needed by many
> > users we talked to.  There are two flavors of AutoScalingGroup resources in
> > Heat today: the AWS-based one and the Heat native one.  As more requests
> > coming in, the team has proposed to separate auto-scaling support into a
> > separate service so that people who are interested in it can jump onto it.  At
> > the same time, Heat engine (especially the resource type code) will be
> > drastically simplified.  The separated AS service could move forward more
> > rapidly and efficiently.
> > 
> > This work was proposed a while ago with the following wiki and blueprints
> > (mostly approved during Havana cycle), but the progress is slow.  A group of
> > developers now volunteer to take over this work and move it forward.
> > 
> > wiki: https://wiki.openstack.org/wiki/Heat/AutoScaling
> > BPs:
> >  - https://blueprints.launchpad.net/heat/+spec/as-lib-db
> >  - https://blueprints.launchpad.net/heat/+spec/as-lib
> >  - https://blueprints.launchpad.net/heat/+spec/as-engine-db
> >  - https://blueprints.launchpad.net/heat/+spec/as-engine
> >  - https://blueprints.launchpad.net/heat/+spec/autoscaling-api
> >  - https://blueprints.launchpad.net/heat/+spec/autoscaling-api-client
> >  - https://blueprints.launchpad.net/heat/+spec/as-api-group-resource
> >  - https://blueprints.launchpad.net/heat/+spec/as-api-policy-resource
> >  - https://blueprints.launchpad.net/heat/+spec/as-api-webhook-trigger-
> > resource
> >  - https://blueprints.launchpad.net/heat/+spec/autoscaling-api-resources
> > 
> > Once this whole thing lands, Heat engine will talk to the AS engine in terms of
> > ResourceGroup, ScalingPolicy, Webhooks.  Heat engine won't care how auto-
> > scaling is implemented although the AS engine may in turn ask Heat to
> > create/update stacks for scaling's purpose.  In theory, AS engine can
> > create/destroy resources by directly invoking other OpenStack services.  This
> > new AutoScaling service may eventually have its own DB, engine, API, api-
> > client.  We can definitely aim high while work hard on real code.
> > 
> > After reviewing the BPs/Wiki and some communication, we get two options
> > to push forward this.  I'm writing this to solicit ideas and comments from the
> > community.
> > 
> > Option A: Top-Down Quick Split
> > ------------------------------
> 
> Do you want to drop support of AS from heat altogether? Many people would disagree with drop of AS (even drop of HARestarter is problem). We don't really want to support duplicate systems, so having 2 engines of autoscalling would be wrong.
> That being said, I can see big gap which heat (or services around) could fill - intelligent orchiestration. By that I mean autohealing, auto-redeploying, autoscalling and pretty much auto-whatever. Clouds are fluid, we could provide framework for that. Heat would be great tool for that because it has context of whole stack, and in fact all we do would be stack update.

Michael, no.  Heat will continue to support AS as it does today, though
the underlying implementation will be drastically changed.  There won't
be two engines of AUTOSCALING, but one Heat engine focusing on
orchestration and another AS engine focusing on autoscaling.

> > This means we will follow a roadmap shown below, which is not 100%
> > accurate yet and very rough:
> > 
> >   1) Get the separated REST service in place and working
> >   2) Switch Heat resources to use the new REST service
> > 
> > Pros:
> >   - Separate code base means faster review/commit cycle
> >   - Less code churn in Heat
> > Cons:
> >   - A new service need to be installed/configured/launched
> >   - Need commitments from dedicated, experienced developers from very
> >     beginning
> > 
> > Option B: Bottom-Up Slow Growth
> > -------------------------------
> 
> Personally I'd be advocate of fixing what we have instead of making new thing. Maybe we should make it separate process (as long as we'll try to keep consistent api)? Maybe add place for new logic (autohealing?), but still keep that inside heat.
> One thing - we'll need to make concurrent updates really robust when we want to make whole thing automatic (I'm talking about convergence).

So basically, you opt to option B, if I'm understanding this correctly.
:)  
I believe the convergence work (especially the parallelization chapter)
is orthogonal to autoscaling. 


> > The roadmap is more conservative, with many (yes, many) incremental
> > patches to migrate things carefully.
> > 
> >   1) Separate some of the autoscaling logic into libraries in Heat
> >   2) Augment heat-engine with new AS RPCs
> >   3) Switch AS related resource types to use the new RPCs
> >   4) Add new REST service that also talks to the same RPC
> >      (create new GIT repo, API endpoint and client lib...)
> > 
> > Pros:
> >   - Less risk breaking user lands with each revision well tested
> >   - More smooth transition for users in terms of upgrades
> > 
> > Cons:
> >   - A lot of churn within Heat code base, which means long review cycles
> >   - Still need commitments from cores to supervise the whole process
> > 
> > There could be option C, D... but the two above are what we came up with
> > during the discussion.
> > 
> > Another important thing we talked about is about the open discussion on
> > this.  OpenStack Wiki seems a good place to document settled designs but
> > not for interactive discussions.  Probably we should leverage etherpad and
> > the mailinglist when moving forward.  Suggestions on this are also welcomed.
> 
> Wouldn't series of specs be better+wiki/etherpad to rule them all?
> 
> > Thanks.
> > 
> > Regards,
> >  Qiming
> >