Open Stack

Let me elaborate a little on my thoughts about software orchestration, and 
respond to the recent mails from Zane and Debo.  I have expanded my 
picture at 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and added a companion picture at 
https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g 
that shows an alternative.

One of the things I see going on is discussion about better techniques for 
software orchestration than are supported in plain CFN.  Plain CFN allows 
any script you want in userdata, and prescription of certain additional 
setup elsewhere in cfn metadata.  But it is all mixed together and very 
concrete.  I think many contributors would like to see something with more 
abstraction boundaries, not only within one template but also the ability 
to have modular sources.

I work closely with some colleagues who have a particular software 
orchestration technology they call Weaver.  It takes as input for one 
deployment not a single monolithic template but rather a collection of 
modules.  Like higher level constructs in programming languages, these 
have some independence and can be re-used in various combinations and 
ways.  Weaver has a compiler that weaves together the given modules to 
form a monolithic model.  In fact, the input is a modular Ruby program, 
and the Weaver compiler is essentially running that Ruby program; this 
program produces the monolithic model as a side effect.  Ruby is a pretty 
good language in which to embed a domain-specific language, and my 
colleagues have done this.  The modular Weaver input mostly looks 
declarative, but you can use Ruby to reduce the verboseness of, e.g., 
repetitive stuff --- as well as plain old modularity with abstraction.  We 
think the modular Weaver input is much more compact and better for human 
reading and writing than plain old CFN.  This might not be obvious when 
you are doing the "hello world" example, but when you get to realistic 
examples it becomes clear.

The Weaver input discusses infrastructure issues, in the rich way Debo and 
I have been advocating, as well as software.  For this reason I describe 
it as an integrated model (integrating software and infrastructure 
issues).  I hope for HOT to evolve to be similarly expressive to the 
monolithic integrated model produced by the Weaver compiler.

In Weaver, as well as in some of the other software orchestration 
technologies being discussed, there is a need for some preparatory work 
before the infrastructure (e.g., VMs) is created.  This preparatory stage 
begins the implementation of the software orchestration abstractions. Here 
is the translation from something more abstract into flat userdata and 
other cfn metadata.  For Weaver, this stage also involves some 
stack-specific setup in a distinct coordination service.  When the VMs 
finally run their userdata, the Weaver-generated scripts there use that 
pre-configured part of the coordination service to interact properly with 
each other.

I think that, to a first-order approximation, the software orchestration 
preparatory stage commutes with holistic infrastructure scheduling.  They 
address independent issues, and can be done in either order.  That is why 
I have added a companion picture; the two pictures show the two orders.

My claim of commutativity is limited, as I and colleagues have 
demonstrated only one of the two orderings; the other is just a matter of 
recent thought.  There could be gotchas lurking in there.

Between the two orderings, I have a preference for the one I first 
mentioned and have experience with actually running.  It has the virtue of 
keeping related things closer together: the software orchestration 
compiler is next to the software orchestration preparatory stage, and the 
holistic infrastructure scheduling is next to the infrastructure 
orchestration.

In response to Debo's remark about flexibility: I am happy to see an 
architecture that allows either ordering if it turns out that they are 
both viable and the community really wants that flexibility.  I am not so 
sure we can totally give up on architecting where things go, but this 
level of flexibility I can understand and get behind (provided it works).

Just as a LP solver is a general utility whose uses do not require 
architecting, I can imagine a higher level utility that solves abstract 
placement problems.  Actually, this is not a matter of imagination.  My 
group has been evolving such a thing for years.  It is now based, as Debo 
recommends, on a very flexible and general optimization algorithm.  But 
the plumbing between it and the rest of the system is significant; I would 
not expect many users to take on that magnitude of task.

I do not really want to get into dogmatic fights over what gets labelled 
"heat".  I will leave the questions about which piece goes where in the 
OpenStack programs and projects to those more informed and anointed.  What 
I am trying to accomplish is an agreement about how various bits of 
functionality will relate.

Zane wrote:
> To take the first example, wouldn't your holistic scheduler effectively 
have
> to reserve a compute instance and some directly attached block storage 
prior
> to actually creating them? Have you considered Climate rather than Heat 
as
> an integration point?

I had not considered Climate.  Based on recent ML traffic, I see that 
Climate is about scheduling into the future, whereas I am only trying to 
talk about scheduling for the present.  OTOH, perhaps you are concerned 
about concurrency issues.  I am too.  Doing a better job on that is a big 
part of the revision my group is working on now.  I think it can be done. 
I plan to post a pointer to some details soon.

Perhaps the concern is about competition between two managers trying to 
manage the same resources.  I think that is (a) something that can not be 
completely avoided and (b) impossible to do well.  My preference is to 
focus on one manager, and make sure it tolerates surprises in a way that 
is not terrible.  Even without competing managers, bugs and other 
unexpected failures will cause nasty surprises.

In response to my remark about infrastructure orchestration being 
downstream from holistic infrastructure scheduling, Zane wrote:
> I agree that it's necessarily 'downstream' (in the sense of happening
> afterwards). I'd hesitate to use the word 'logically', since I think by 
it's
> very nature a holistic scheduler introduces dependencies between 
services
> that were intended to be _logically_ independent.

I am not sure I understand the objection here.  All I am saying is that a 
holistic infrastructure scheduler is one of the things that needs the 
services of something that does infrastructure orchestration, and I see no 
need to introduce a new such thing.  Surely it is not bad for various 
things to use the existing mechanism.  I see plans for Trove to also do 
this, and for the revised autoscaling service to also do this.

Zane later wrote:
> As proposed, the software configs contain directives like 'hosted_on:
> server_name'. (I don't know that I'm a huge fan of this design, but I 
don't
> think the exact details are relevant in this context.) There's no
> non-trivial processing in the preparatory stage of software 
orchestration
> that would require it to be performed before scheduling could occur.

I hope I have addressed that with my remarks above about software 
orchestration.

Zane also wrote:
> Let's make sure we distinguish between doing holistic scheduling, which
> requires a priori knowledge of the resources to be created, and 
automatic
> scheduling, which requires psychic knowledge of the user's mind. (Did 
the
> user want to optimise for performance or availability? How would you 
infer
> that from the template?)

One reason I favor holistic infrastructure scheduling is that I want its 
input to be richer than today's CFN templates.  Like Debo, I think the 
input can contain the kind of information that would otherwise require 
mind-reading.  My group has been working examples involving multiple 
levels of anti-co-location statements, network reachability and proximity 
statements, disk exclusivity statements, and statements about the presence 
of licensed products.

Zane ended with:
> .. There's nothing that happens while preparing the
> software configurations that's necessary for the former nor sufficient 
for
> the latter.

If I understand correctly, I agree, as mentioned in my above remarks about 
how software orchestration preparation commutes with holistic 
infrastructure scheduling.

Regards,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130925/2b0a665a/attachment.html>