Open Stack

Excellent discussion on various issues around orchestration and
coordination -- thanks to you all, in particular to Clint, Angus, Stan,
Thomas, Joshua, Zane, Steve ...

After reading the discussions, I am finding the following themes emerging
(please feel free to correct/add):

1. Most of the building blocks needed for effective coordination and
orchestration are already in Heat/HOT.
2. Heat would like to view software configuration as a resource (type) with
related providers + plugins
3. There is scope for communication/synchronization mechanisms that would
complement the wait-conditions and signals

I would like to propose a simple abstraction that would complement the
current wait-conditions and signals. My proposal is based our experience
with supporting such an abstraction on our DSL and also on an extension of
Heat.  In a nut-shell, this abstraction is a global data space (visible
across resources, stacks) from which resources can read and write their
inputs / outputs PLUS the semantics that reads will block until the read
values are available and writes are non-blocking. We used ZooKeeper to
implement this global data space and the blocking-read/non-blocking-writes
semantics. But, these could be implemented using several other mechanisms
and I believe the techniques currently used by Heat for meta-data service
can be used here.

I would like to make clear that I am not proposing a replacement for
wait-conditions and signals. I am hoping that wait-conditions and signals
would be used by power-users (concurrent/distributed programming experts)
and the proposed abstraction would be used by folks (like me) who do not
want to reason about concurrency and related problems. Also, the proposed
global data-space with blocking reads and non-blocking writes is not a new
idea (google tuple-spaces, linda) and it has been proven in other domains
such as coordination languages to improve the level of abstraction and
productivity.

The benefits of the proposed abstraction are:
G1. Support finer granularity of dependences
G2. Allow Heat to reason/analyze about these dependences so that it can
order resource creations/management
G3. Avoid classic synchronization problems such as dead-locks and race
conditions
G4 *Conjecture* : Capture most of the coordination use cases (including
those required for software configuration / orchestration).

Here is more detailed description: Let us say that we can use either
pre-defined or custom resource types to define resources at arbitrary
levels of granularity. This can be easily supported and I guess is already
possible in current version of Heat/HOT. Given this, the proposed
abstraction has two parts: (1) an interface style specification a
resource's inputs and outputs and (2) a global name/data space. The
interface specification which would capture

- INPUTS: all the attributes that are consumed/used/read by that resource
(currently, we have Ref, GetAttrs that can give this implicitly)

- OUTPUTS: all the attributes that are produced/written by that resource (I
do not know if this write-set is currently well-defined for a resource. I
think some of them are implicitly defined by Heat on particular resource
types.)

- Global name-space and data-space : all the values produced and consumed
(INPUTS/OUTPUTS) are described using a names that are fully qualified
(XXX.stack_name.resource_name.property_name). The data values associated
with these names are stored in a global data-space.  Reads are blocking,
i.e., reading a value will block the execution resource/thread until the
value is available. Writes are non-blocking, i.e., any thread can write a
value and the write will succeed immediately.

The ability to define resources at arbitrary levels of granularity together
with the explicit specification of INPUTS/OUTPUTS allows us to reap the
benefits G1 and G2 outlined above. Note that the ability to reason about
the inputs/outputs of each resource and the induced dependencies will also
allow Heat to detect dead-locks via dependence cycles (benefit G3). This is
already done today in Heat for Refs, GetAttr on base-resources, but the
proposal is to extend the same to arbitrary attributes for any resource.
The blocking-read and non-blocking writes further structures the
specification to avoid deadlocks and race conditions (benefit G3).

As for G4, the conjecture, I can only give as evidence our experience with
using our DSL with the proposed abstraction to deploy a few reasonably
large applications :-)

I would like to know your comments and suggestions. Also, if there is
interest I can write a Blueprint / proposal with more details and
use-cases.

Thanks,
LN



Clint Byrum <clint at fewbar.com> wrote on 10/11/2013 12:40:19 PM:

> From: Clint Byrum <clint at fewbar.com>
> To: openstack-dev <openstack-dev at lists.openstack.org>
> Date: 10/11/2013 12:43 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700:
> > Hello,
> >
> > Thanks Angus, Clint, I've got your design.
> >
> > It seems that Murano can built on top of that. With service metadata
> > knowledge Murano can generate HOT templates with set of interdependent
> > configs.
> > Here is what will be needed:
> >
> > 1. Ability to implement support for custom software configuration tool
> > (type: OS::SoftwareConfig::MuranoAgent)
>
> These are really syntactic sugar, but you can implement them as resource
> plugins for users who want Murano resources. In the absence of this,
> just putting things in the free form metadata area of the resource that
> the MuranoAgent can interpret would suffice.
>
> > 2. Ability to provide arbitrary input values for the config
>
> We already have that, there's a free-form json document called "metadata"
> attached to every resource. Or maybe I missed what you mean here. The
> new capability that is in the works that will make that better is to
> have multiple reusable metadata blocks referenced on one instance.
>
> > 3. Ability to return arbitrary (JSON-compatible) data structure from
config
> > application and use attributes of that structure as an input for other
> > configs
>
> Note that I'd like to see more use cases specified for this ability. The
> random string generator that Steve Baker has put up should handle most
> cases where you just need passwords. Generated key sharing might best
> be deferred to something like Barbican which does a lot more than Heat
> to try and keep your secrets safe.
>
> > 4. Ability to provide config body that is an input to Murano Agent of
> > arbitrary size
>
> Isn't this the same as 2?
>
> > 5. Work well with large graph of configs with a lot of dependencies.
> > Independent configs on different VMs should be applied in parallel.
> >
>
> Yes this does look good. For dependent configs, the exsiting wait
> condition can be used.
>
> > Does it confirm to your plans?
> >
>
> I think it confirms that we're heading toward consensus on where to draw
> the software config vs. infrastructure orchestration line. That is very
> exciting. :)
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131011/c1a0b063/attachment.html>