Open Stack

Excerpts from Mike Spreitzer's message of 2013-09-24 22:03:21 -0700:
> Let me elaborate a little on my thoughts about software orchestration, and 
> respond to the recent mails from Zane and Debo.  I have expanded my 
> picture at 
> https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
> and added a companion picture at 
> https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g 
> that shows an alternative.
> 
> One of the things I see going on is discussion about better techniques for 
> software orchestration than are supported in plain CFN.  Plain CFN allows 
> any script you want in userdata, and prescription of certain additional 
> setup elsewhere in cfn metadata.  But it is all mixed together and very 
> concrete.  I think many contributors would like to see something with more 
> abstraction boundaries, not only within one template but also the ability 
> to have modular sources.
> 

Yes please. Orchestrate things, don't configure them. That is what
configuration tools are for.

There is a third stealth-objective that CFN has caused to linger in
Heat. That is "packaging cloud applications". By allowing the 100%
concrete CFN template to stand alone, users can "ship" the template.

IMO this marrying of software assembly, config, and orchestration is a
concern unto itself, and best left outside of the core infrastructure
orchestration system.

> I work closely with some colleagues who have a particular software 
> orchestration technology they call Weaver.  It takes as input for one 
> deployment not a single monolithic template but rather a collection of 
> modules.  Like higher level constructs in programming languages, these 
> have some independence and can be re-used in various combinations and 
> ways.  Weaver has a compiler that weaves together the given modules to 
> form a monolithic model.  In fact, the input is a modular Ruby program, 
> and the Weaver compiler is essentially running that Ruby program; this 
> program produces the monolithic model as a side effect.  Ruby is a pretty 
> good language in which to embed a domain-specific language, and my 
> colleagues have done this.  The modular Weaver input mostly looks 
> declarative, but you can use Ruby to reduce the verboseness of, e.g., 
> repetitive stuff --- as well as plain old modularity with abstraction.  We 
> think the modular Weaver input is much more compact and better for human 
> reading and writing than plain old CFN.  This might not be obvious when 
> you are doing the "hello world" example, but when you get to realistic 
> examples it becomes clear.
> 
> The Weaver input discusses infrastructure issues, in the rich way Debo and 
> I have been advocating, as well as software.  For this reason I describe 
> it as an integrated model (integrating software and infrastructure 
> issues).  I hope for HOT to evolve to be similarly expressive to the 
> monolithic integrated model produced by the Weaver compiler.
> 

Indeed, we're dealing with this very problem in TripleO right now. We need
to be able to compose templates that vary slightly for various reasons.

A ruby DSL is not something I think is ever going to happen in
OpenStack. But python has its advantages for DSL as well. I have been
trying to use clever tricks in yaml for a while, but perhaps we should
just move to a client-side python DSL that pushes the compiled yaml/json
templates into the engine.