Open Stack

Excerpts from Thomas Spatzier's message of 2013-09-25 00:59:44 -0700:
> Clint Byrum <clint at fewbar.com> wrote on 25.09.2013 08:46:57:
> > From: Clint Byrum <clint at fewbar.com>
> > To: openstack-dev <openstack-dev at lists.openstack.org>,
> > Date: 25.09.2013 08:48
> > Subject: Re: [openstack-dev] [heat] [scheduler] Bringing things
> > together for Icehouse (now featuring software orchestration)
> >
> > Excerpts from Mike Spreitzer's message of 2013-09-24 22:03:21 -0700:
> > > Let me elaborate a little on my thoughts about software orchestration,
> and
> > > respond to the recent mails from Zane and Debo.  I have expanded my
> > > picture at
> > > https://docs.google.com/drawings/d/
> > 1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U
> > > and added a companion picture at
> > > https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-
> > GQQ1bRVgBpJdstpu0lH_TONw6g
> > > that shows an alternative.
> > >
> > > One of the things I see going on is discussion about better techniques
> for
> > > software orchestration than are supported in plain CFN.  Plain CFN
> allows
> > > any script you want in userdata, and prescription of certain additional
> 
> > > setup elsewhere in cfn metadata.  But it is all mixed together and very
> 
> > > concrete.  I think many contributors would like to see something with
> more
> > > abstraction boundaries, not only within one template but also the
> ability
> > > to have modular sources.
> > >
> >
> > Yes please. Orchestrate things, don't configure them. That is what
> > configuration tools are for.
> >
> > There is a third stealth-objective that CFN has caused to linger in
> > Heat. That is "packaging cloud applications". By allowing the 100%
> > concrete CFN template to stand alone, users can "ship" the template.
> >
> > IMO this marrying of software assembly, config, and orchestration is a
> > concern unto itself, and best left outside of the core infrastructure
> > orchestration system.
> >
> > > I work closely with some colleagues who have a particular software
> > > orchestration technology they call Weaver.  It takes as input for one
> > > deployment not a single monolithic template but rather a collection of
> > > modules.  Like higher level constructs in programming languages, these
> > > have some independence and can be re-used in various combinations and
> > > ways.  Weaver has a compiler that weaves together the given modules to
> > > form a monolithic model.  In fact, the input is a modular Ruby program,
> 
> > > and the Weaver compiler is essentially running that Ruby program; this
> > > program produces the monolithic model as a side effect.  Ruby is a
> pretty
> > > good language in which to embed a domain-specific language, and my
> > > colleagues have done this.  The modular Weaver input mostly looks
> > > declarative, but you can use Ruby to reduce the verboseness of, e.g.,
> > > repetitive stuff --- as well as plain old modularity with abstraction.
> We
> > > think the modular Weaver input is much more compact and better for
> human
> > > reading and writing than plain old CFN.  This might not be obvious when
> 
> > > you are doing the "hello world" example, but when you get to realistic
> > > examples it becomes clear.
> > >
> > > The Weaver input discusses infrastructure issues, in the rich way Debo
> and
> > > I have been advocating, as well as software.  For this reason I
> describe
> > > it as an integrated model (integrating software and infrastructure
> > > issues).  I hope for HOT to evolve to be similarly expressive to the
> > > monolithic integrated model produced by the Weaver compiler.
> 
> I don't fully get this idea of HOT consuming a monolithic model produced by
> some compiler - be it Weaver or anything else.
> I thought the goal was to develop HOT in a way that users can actually
> write HOT, as opposed to having to use some "compiler" to produce some
> useful model.
> So wouldn't it make sense to make sure we add the right concepts to HOT to
> make sure we are able to express what we want to express and have things
> like composability, re-use, substitutability?
> 

We saw this in the history of puppet in fact, where the DSL was always the
problem when trying to make less-than-obvious components, and eventually
puppet had to grow a full "ruby dsl" to avoid those mistakes and keep up
with Chef's language-first approach.

> > >
> >
> > Indeed, we're dealing with this very problem in TripleO right now. We
> need
> > to be able to compose templates that vary slightly for various reasons.
> >
> > A ruby DSL is not something I think is ever going to happen in
> > OpenStack. But python has its advantages for DSL as well. I have been
> > trying to use clever tricks in yaml for a while, but perhaps we should
> > just move to a client-side python DSL that pushes the compiled yaml/json
> > templates into the engine.
> 
> As said in my comment above, I would like to see us focusing on the
> agreement of one language - HOT - instead of yet another DSL.
> There are things out there that are well established (like chef or puppet),
> and HOT should be able to efficiently and intuitively use those things and
> orchestrate components built using those things.
> 
> Anyway, this might be off the track that was originally discussed in this
> thread (i.e. holistic scheduling and so on) ...
>

So yes, we should make sure to interface with other tools. But it stands
to reason that not all orchestration solutions are representable as a
series of data structures. We need loops, and conditionals, and callables,
so that we can write reusable, testable components. Fighting that just
means punting it off to a compiler anyway.

Orchestration DSL authors are not immune to the sort of pitfalls any
other language authors may encounter, so we shouldn't consider the
problem space special and ignore the principles other languages have
embraced to avoid the dark scary spots.

I think we already have some summit suggestions for discussing HOT,
it would be good to come prepared with some visions for the future
of HOT so that we can hash these things out, so I'd like to see this
discussion continue.