Open Stack

Hi Joshua,

I agree that there is an element of taskflow in what I described. But, I am
aiming for something much more lightweight which can be naturally blended
with HOT constructs and Heat engine. To be a bit more specific, Heat
already has dependencies and coordination mechanisms. So, I am aiming for
may be just one additional construct in Heat/HOT and some logic in Heat
that would support coordination.

Thanks,
LN

_________________________________________________________
Lakshminarayanan Renganarayana
Research Staff Member
IBM T.J. Watson Research Center
http://researcher.ibm.com/person/us-lrengan


Joshua Harlow <harlowja at yahoo-inc.com> wrote on 10/09/2013 03:55:00 PM:

> From: Joshua Harlow <harlowja at yahoo-inc.com>
> To: OpenStack Development Mailing List <openstack-
> dev at lists.openstack.org>, Lakshminaraya Renganarayana/Watson/IBM at IBMUS
> Date: 10/09/2013 03:55 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Your example sounds a lot like what taskflow is build for doing.
>
> https://github.com/stackforge/taskflow/blob/master/taskflow/
> examples/calculate_in_parallel.py is a decent example.
>
> In that one, tasks are created and input/output dependencies are
> specified (provides, rebind, and the execute function arguments itself).
>
> This is combined into the taskflow concept of a flow, one of those
> flows types is a dependency graph.
>
> Using a parallel engine (similar in concept to a heat engine) we can
> run all non-dependent tasks in parallel.
>
> An example that I just created that shows this (and shows it
> running) that closer matches your example.
>
> Program (this will work against the current taskflow codebase):
> http://paste.openstack.org/show/48156/
>
> Output @ http://paste.openstack.org/show/48157/
>
> -Josh
>
> From: Lakshminaraya Renganarayana <lrengan at us.ibm.com>
> Reply-To: OpenStack Development Mailing List <openstack-
> dev at lists.openstack.org>
> Date: Wednesday, October 9, 2013 11:31 AM
> To: OpenStack Development Mailing List
<openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Steven Hardy <shardy at redhat.com> wrote on 10/09/2013 05:24:38 AM:
>
> >
> > So as has already been mentioned, Heat defines an internal workflow,
based
> > on the declarative model defined in the template.
> >
> > The model should define dependencies, and Heat should convert those
> > dependencies into a workflow internally.  IMO if the user also needs to
> > describe a workflow explicitly in the template, then we've probably
failed
> > to provide the right template interfaces for describing depenendencies.
>
> I agree with Steven here, models should define the dependencies and Heat
> should realize/enforce them. An important design issue is granularity at
> which dependencies are defined and enforced. I am aware of the
wait-condition
> and signal constructs in Heat, but I find them a bit low-level as
> they are prone
> to the classic dead-lock and race condition problems.  I would like to
have
> higher level constructs that support finer-granularity dependences which
> are needed for software orchestration. Reading through the various
disucssion
> on this topic in this mailing list, I see that many would like to have
such
> higher level constructs for coordination.
>
> In our experience with software orchestration using our own DSL and also
with
> some extensions to Heat, we found that the granularity of VMs or
> Resources to be
> too coarse for defining dependencies for software orchestration. For
> example, consider
> a two VM app, with VMs vmA, vmB, and a set of software components
> (ai's and bi's)
> to be installed on them:
>
> vmA = base-vmA + a1 + a2 + a3
> vmB = base-vmB + b1 + b2 + b3
>
> let us say that software component b1 of vmB, requires a config
> value produced by
> software component a1 of vmA. How to declaratively model this
> dependence? Clearly,
> modeling a dependence between just base-vmA and base-vmB is not
> enough. However,
> defining a dependence between the whole of vmA and vmB is too
> coarse. It would be ideal
> to be able to define a dependence at the granularity of software
> components, i.e.,
> vmB.b1 depends on vmA.a1. Of course, it would also be good to
> capture what value
> is passed between vmB.b1 and vmA.a1, so that the communication can
> be facilitated
> by the orchestration engine.
>
> We found that such finer granular modeling of the dependencies
> provides two valuable benefits:
>
> 1. Faster total (resources + software setup) deployment time. For
> the example described
> above, a coarse-granularity dependence enforcer would start the
> deployment of base-vmB after
> vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence
> enforcer would start base-vmA
> and base-vmB concurrently, and then suspend the execution of vmB.b1
> until vmA.a1 is complete and then
> let the rest of deployment proceed concurrently, resulting in a
> faster completion.
>
> 2. More flexible dependencies. For example, mutual dependencies
> between resources,
> which can be satisfied when orchestrated at a finer granularity.
> Using the example described
> above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and
> also vmA.a3 depends_on vmB.b2,
> but coarse-granularity model would flag this as a cyclic dependence.
>
> There are two aspects that needs support:
>
> 1. Heat/HOT template level constructs to support declarative
> expression of such fine-granularity
> dependencies and the values communicated / passed for the dependence.
> 2. Support from Heat engine / analyzer in supporting the runtime
> ordering, coordination between
> resources, and also the communication of the values.
>
> What are your thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131009/a071646c/attachment-0001.html>