<html><body>
<p><font size="2" face="sans-serif">Hi Joshua,</font><br>
<br>
<font size="2" face="sans-serif">I agree that there is an element of taskflow in what I described. But, I am aiming for something much more lightweight which can be naturally blended with HOT constructs and Heat engine. To be a bit more specific, Heat already has dependencies and coordination mechanisms. So, I am aiming for may be just one additional construct in Heat/HOT and some logic in Heat that would support coordination. </font><br>
<br>
<font size="2" face="sans-serif">Thanks,</font><br>
<font size="2" face="sans-serif">LN</font><br>
<br>
<font size="2" face="sans-serif">_________________________________________________________<br>
Lakshminarayanan Renganarayana<br>
Research Staff Member<br>
IBM T.J. Watson Research Center<br>
<a href="http://researcher.ibm.com/person/us-lrengan">http://researcher.ibm.com/person/us-lrengan</a><br>
</font><br>
<br>
<tt><font size="2">Joshua Harlow <harlowja@yahoo-inc.com> wrote on 10/09/2013 03:55:00 PM:<br>
<br>
> From: Joshua Harlow <harlowja@yahoo-inc.com></font></tt><br>
<tt><font size="2">> To: OpenStack Development Mailing List <openstack-<br>
> dev@lists.openstack.org>, Lakshminaraya Renganarayana/Watson/IBM@IBMUS</font></tt><br>
<tt><font size="2">> Date: 10/09/2013 03:55 PM</font></tt><br>
<tt><font size="2">> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration <br>
> proposal for workflows</font></tt><br>
<tt><font size="2">> <br>
> Your example sounds a lot like what taskflow is build for doing.</font></tt><br>
<tt><font size="2">> <br>
> <a href="https://github.com/stackforge/taskflow/blob/master/taskflow/">https://github.com/stackforge/taskflow/blob/master/taskflow/</a><br>
> examples/calculate_in_parallel.py is a decent example.</font></tt><br>
<tt><font size="2">> <br>
> In that one, tasks are created and input/output dependencies are <br>
> specified (provides, rebind, and the execute function arguments itself).</font></tt><br>
<tt><font size="2">> <br>
> This is combined into the taskflow concept of a flow, one of those <br>
> flows types is a dependency graph.</font></tt><br>
<tt><font size="2">> <br>
> Using a parallel engine (similar in concept to a heat engine) we can<br>
> run all non-dependent tasks in parallel.</font></tt><br>
<tt><font size="2">> <br>
> An example that I just created that shows this (and shows it <br>
> running) that closer matches your example.</font></tt><br>
<tt><font size="2">> <br>
> Program (this will work against the current taskflow codebase): <br>
> <a href="http://paste.openstack.org/show/48156/">http://paste.openstack.org/show/48156/</a></font></tt><br>
<tt><font size="2">> <br>
> Output @ <a href="http://paste.openstack.org/show/48157/">http://paste.openstack.org/show/48157/</a></font></tt><br>
<tt><font size="2">> <br>
> -Josh</font></tt><br>
<tt><font size="2">> <br>
> From: Lakshminaraya Renganarayana <lrengan@us.ibm.com><br>
> Reply-To: OpenStack Development Mailing List <openstack-<br>
> dev@lists.openstack.org><br>
> Date: Wednesday, October 9, 2013 11:31 AM<br>
> To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org><br>
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration <br>
> proposal for workflows</font></tt><br>
<tt><font size="2">> <br>
> Steven Hardy <shardy@redhat.com> wrote on 10/09/2013 05:24:38 AM:<br>
> <br>
> > <br>
> > So as has already been mentioned, Heat defines an internal workflow, based<br>
> > on the declarative model defined in the template.<br>
> > <br>
> > The model should define dependencies, and Heat should convert those<br>
> > dependencies into a workflow internally. IMO if the user also needs to<br>
> > describe a workflow explicitly in the template, then we've probably failed<br>
> > to provide the right template interfaces for describing depenendencies.<br>
> <br>
> I agree with Steven here, models should define the dependencies and Heat<br>
> should realize/enforce them. An important design issue is granularity at<br>
> which dependencies are defined and enforced. I am aware of the wait-condition<br>
> and signal constructs in Heat, but I find them a bit low-level as <br>
> they are prone<br>
> to the classic dead-lock and race condition problems. I would like to have <br>
> higher level constructs that support finer-granularity dependences which<br>
> are needed for software orchestration. Reading through the various disucssion<br>
> on this topic in this mailing list, I see that many would like to have such<br>
> higher level constructs for coordination.<br>
> <br>
> In our experience with software orchestration using our own DSL and also with <br>
> some extensions to Heat, we found that the granularity of VMs or <br>
> Resources to be <br>
> too coarse for defining dependencies for software orchestration. For<br>
> example, consider <br>
> a two VM app, with VMs vmA, vmB, and a set of software components <br>
> (ai's and bi's) <br>
> to be installed on them:<br>
> <br>
> vmA = base-vmA + a1 + a2 + a3<br>
> vmB = base-vmB + b1 + b2 + b3<br>
> <br>
> let us say that software component b1 of vmB, requires a config <br>
> value produced by<br>
> software component a1 of vmA. How to declaratively model this <br>
> dependence? Clearly,<br>
> modeling a dependence between just base-vmA and base-vmB is not <br>
> enough. However,<br>
> defining a dependence between the whole of vmA and vmB is too <br>
> coarse. It would be ideal<br>
> to be able to define a dependence at the granularity of software <br>
> components, i.e., <br>
> vmB.b1 depends on vmA.a1. Of course, it would also be good to <br>
> capture what value <br>
> is passed between vmB.b1 and vmA.a1, so that the communication can <br>
> be facilitated<br>
> by the orchestration engine. <br>
> <br>
> We found that such finer granular modeling of the dependencies <br>
> provides two valuable benefits:<br>
> <br>
> 1. Faster total (resources + software setup) deployment time. For <br>
> the example described<br>
> above, a coarse-granularity dependence enforcer would start the <br>
> deployment of base-vmB after<br>
> vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence <br>
> enforcer would start base-vmA<br>
> and base-vmB concurrently, and then suspend the execution of vmB.b1 <br>
> until vmA.a1 is complete and then <br>
> let the rest of deployment proceed concurrently, resulting in a <br>
> faster completion.<br>
> <br>
> 2. More flexible dependencies. For example, mutual dependencies <br>
> between resources, <br>
> which can be satisfied when orchestrated at a finer granularity. <br>
> Using the example described<br>
> above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and <br>
> also vmA.a3 depends_on vmB.b2,<br>
> but coarse-granularity model would flag this as a cyclic dependence.<br>
> <br>
> There are two aspects that needs support:<br>
> <br>
> 1. Heat/HOT template level constructs to support declarative <br>
> expression of such fine-granularity<br>
> dependencies and the values communicated / passed for the dependence.<br>
> 2. Support from Heat engine / analyzer in supporting the runtime <br>
> ordering, coordination between<br>
> resources, and also the communication of the values. <br>
> <br>
> What are your thoughts? </font></tt></body></html>