Open Stack

Your example sounds a lot like what taskflow is build for doing.

https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py is a decent example.

In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself).

This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph.

Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel.

An example that I just created that shows this (and shows it running) that closer matches your example.

Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/

Output @ http://paste.openstack.org/show/48157/

-Josh

From: Lakshminaraya Renganarayana <lrengan at us.ibm.com<mailto:lrengan at us.ibm.com>>
Reply-To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Wednesday, October 9, 2013 11:31 AM
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows


Steven Hardy <shardy at redhat.com<mailto:shardy at redhat.com>> wrote on 10/09/2013 05:24:38 AM:

>
> So as has already been mentioned, Heat defines an internal workflow, based
> on the declarative model defined in the template.
>
> The model should define dependencies, and Heat should convert those
> dependencies into a workflow internally.  IMO if the user also needs to
> describe a workflow explicitly in the template, then we've probably failed
> to provide the right template interfaces for describing depenendencies.

I agree with Steven here, models should define the dependencies and Heat
should realize/enforce them. An important design issue is granularity at
which dependencies are defined and enforced. I am aware of the wait-condition
and signal constructs in Heat, but I find them a bit low-level as they are prone
to the classic dead-lock and race condition problems.  I would like to have
higher level constructs that support finer-granularity dependences which
are needed for software orchestration. Reading through the various disucssion
on this topic in this mailing list, I see that many would like to have such
higher level constructs for coordination.

In our experience with software orchestration using our own DSL and also with
some extensions to Heat, we found that the granularity of VMs or Resources to be
too coarse for defining dependencies for software orchestration. For example, consider
a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's)
to be installed on them:

vmA = base-vmA + a1 + a2 + a3
vmB = base-vmB + b1 + b2 + b3

let us say that software component b1 of vmB, requires a config value produced by
software component a1 of vmA. How to declaratively model this dependence? Clearly,
modeling a dependence between just base-vmA and base-vmB is not enough. However,
defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal
to be able to define a dependence at the granularity of software components, i.e.,
vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value
is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated
by the orchestration engine.

We found that such finer granular modeling of the dependencies provides two valuable benefits:

1. Faster total (resources + software setup) deployment time. For the example described
above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after
vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA
and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then
let the rest of deployment proceed concurrently, resulting in a faster completion.

2. More flexible dependencies. For example, mutual dependencies between resources,
which can be satisfied when orchestrated at a finer granularity. Using the example described
above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2,
but coarse-granularity model would flag this as a cyclic dependence.

There are two aspects that needs support:

1. Heat/HOT template level constructs to support declarative expression of such fine-granularity
dependencies and the values communicated / passed for the dependence.
2. Support from Heat engine / analyzer in supporting the runtime ordering, coordination between
resources, and also the communication of the values.

What are your thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131009/83aa13fe/attachment.html>