<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Your example sounds a lot like what taskflow is build for doing.</div>
<div><br>
</div>
<div><a href="https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py">https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py</a> is a decent example.</div>
<div><br>
</div>
<div>In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself).</div>
<div><br>
</div>
<div>This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph.</div>
<div><br>
</div>
<div>Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel.</div>
<div><br>
</div>
<div>An example that I just created that shows this (and shows it running) that closer matches your example.</div>
<div><br>
</div>
<div>Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/</div>
<div><br>
</div>
<div>Output @ <a href="http://paste.openstack.org/show/48157">http://paste.openstack.org/show/48157</a>/</div>
<div><br>
</div>
<div>-Josh</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Lakshminaraya Renganarayana <<a href="mailto:lrengan@us.ibm.com">lrengan@us.ibm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 9, 2013 11:31 AM<br>
<span style="font-weight:bold">To: </span>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows<br>
</div>
<div><br>
</div>
<div>
<div>
<p><tt><font size="2">Steven Hardy <<a href="mailto:shardy@redhat.com">shardy@redhat.com</a>> wrote on 10/09/2013 05:24:38 AM:<br>
<br>
> <br>
> So as has already been mentioned, Heat defines an internal workflow, based<br>
> on the declarative model defined in the template.<br>
> <br>
> The model should define dependencies, and Heat should convert those<br>
> dependencies into a workflow internally. IMO if the user also needs to<br>
> describe a workflow explicitly in the template, then we've probably failed<br>
> to provide the right template interfaces for describing depenendencies.<br>
</font></tt><br>
<tt><font size="2">I agree with Steven here, models should define the dependencies and Heat</font></tt><br>
<tt><font size="2">should realize/enforce them. An important design issue is granularity at</font></tt><br>
<tt><font size="2">which dependencies are defined and enforced. I am aware of the wait-condition</font></tt><br>
<tt><font size="2">and signal constructs in Heat, but I find them a bit low-level as they are prone</font></tt><br>
<tt><font size="2">to the classic dead-lock and race condition problems. I would like to have
</font></tt><br>
<tt><font size="2">higher level constructs that support finer-granularity dependences which</font></tt><br>
<tt><font size="2">are needed for software orchestration. Reading through the various disucssion</font></tt><br>
<tt><font size="2">on this topic in this mailing list, I see that many would like to have such</font></tt><br>
<tt><font size="2">higher level constructs for coordination.</font></tt><br>
<br>
<tt><font size="2">In our experience with software orchestration using our own DSL and also with
</font></tt><br>
<tt><font size="2">some extensions to Heat, we found that the granularity of VMs or Resources to be
</font></tt><br>
<tt><font size="2">too coarse for defining dependencies for software orchestration. For example, consider
</font></tt><br>
<tt><font size="2">a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's)
</font></tt><br>
<tt><font size="2">to be installed on them:</font></tt><br>
<br>
<tt><font size="2">vmA = base-vmA + a1 + a2 + a3</font></tt><br>
<tt><font size="2">vmB = base-vmB + b1 + b2 + b3</font></tt><br>
<br>
<tt><font size="2">let us say that software component b1 of vmB, requires a config value produced by</font></tt><br>
<tt><font size="2">software component a1 of vmA. How to declaratively model this dependence? Clearly,</font></tt><br>
<tt><font size="2">modeling a dependence between just base-vmA and base-vmB is not enough. However,</font></tt><br>
<tt><font size="2">defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal</font></tt><br>
<tt><font size="2">to be able to define a dependence at the granularity of software components, i.e.,
</font></tt><br>
<tt><font size="2">vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value
</font></tt><br>
<tt><font size="2">is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated</font></tt><br>
<tt><font size="2">by the orchestration engine. </font></tt><br>
<br>
<tt><font size="2">We found that such finer granular modeling of the dependencies provides two valuable benefits:</font></tt><br>
<br>
<tt><font size="2">1. Faster total (resources + software setup) deployment time. For the example described</font></tt><br>
<tt><font size="2">above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after</font></tt><br>
<tt><font size="2">vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA</font></tt><br>
<tt><font size="2">and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then
</font></tt><br>
<tt><font size="2">let the rest of deployment proceed concurrently, resulting in a faster completion.</font></tt><br>
<br>
<tt><font size="2">2. More flexible dependencies. For example, mutual dependencies between resources,
</font></tt><br>
<tt><font size="2">which can be satisfied when orchestrated at a finer granularity. Using the example described</font></tt><br>
<tt><font size="2">above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2,</font></tt><br>
<tt><font size="2">but coarse-granularity model would flag this as a cyclic dependence.</font></tt><br>
<br>
<tt><font size="2">There are two aspects that needs support:</font></tt><br>
<br>
<tt><font size="2">1. Heat/HOT template level constructs to support declarative expression of such fine-granularity</font></tt><br>
<tt><font size="2">dependencies and the values communicated / passed for the dependence.</font></tt><br>
<tt><font size="2">2. Support from Heat engine / analyzer in supporting the runtime ordering, coordination between</font></tt><br>
<tt><font size="2">resources, and also the communication of the values. </font></tt><br>
<br>
<tt><font size="2">What are your thoughts? </font></tt></p>
</div>
</div>
</span>
</body>
</html>