<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Cool, my thoughts added ;)</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, April 1, 2014 at 2:51 PM<br>
<span style="font-weight:bold">To: </span>Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Cc: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
Even more responses inline :)</div>
<div>
<div>On Mar 31, 2014, at 8:44 PM, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div>More responses inline :)</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Date: </span>Monday, March 31, 2014 at 5:59 PM<br>
<span style="font-weight:bold">To: </span>Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Cc: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Inline...<br>
<div>
<div>On Mar 27, 2014, at 5:10 PM, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">Thanks for the description!</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">The steps here seem very much like what a taskflow engine does (which is good).</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">To connect this to how I think could work in taskflow.</div>
<ol>
<li><font face="Calibri,sans-serif">Someone creates tasks/flows describing the work
<i>to-be-done</i> (converting a DSL -> taskflow tasks/flows/retry[1] objects</font><font face="Calibri,sans-serif">…)</font></li><li><font face="Calibri,sans-serif">On execute(workflow) </font><span style="font-family: Calibri; ">engine creates a new workflow execution, computes the first batch of tasks, creates executor for those tasks (remote, local</span>…) and executes those tasks.</li><li>Waits for response back from <a href="http://docs.python.org/dev/library/concurrent.futures.html">
futures</a> returned from executor.</li><li>Receives futures responses (or receives new response <i>DELAY</i>, for example), or exceptions…</li><li>Continues sending out batches of tasks that can be still be executing (aka tasks that don't have dependency on output of delayed tasks).</li><li>If any delayed tasks after repeating #2-5 as many times as it can, the engine will shut itself down (see
<a href="http://tinyurl.com/l3x3rrb">http://tinyurl.com/l3x3rrb</a>).</li></ol>
</div>
</blockquote>
Why would engine treat long running tasks differently? The model Mistral tried out is the engine sends the batch of tasks and goes asleep; the 'worker/executor' is calling engine back when the task(s) complete. Can it be applied </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Not all ways of using taskflow I think want to go 'asleep', some active conductors I would think actually stay 'awake', that’s why I was thinking this would be a comprimise, the usage of 'DELAY' (via exception or return) would allow a task to say it is
going to be finished sometime in the future (btw started this @ <a href="https://review.openstack.org/#/c/84277">https://review.openstack.org/#/c/84277</a>/ --- WIP, likely won't go into taskflow 0.2 but seems like a reasonable start on something like this
proposal). To me offering just the going asleep way means that all users of taskflow need to have some type of entrypoint (rest, mq, other…) that restarts the engine on result finishing. I'd rather not force that model onto all users (since that seems inappropriate).</div>
</div>
</blockquote>
I understand and agree with not enforcing this model on all users. On a flip side, TaskFlow enforces the current model on Mistral :) It calls for supporting more models (to your earlier point of 3rd mode of operations). The BP proposal is a step in the direction
but still a compromise. How can we support both 'active' and 'passive' models, sharing the same 'library' - of all things responsible for control flow and data flow?</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>To me this is what engines 'types' are for, to support different execution models while using as many of the same components as possible (making usage as seamless as can be).</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><span style="white-space: pre-wrap;">Here is a strawman (</span><span style="white-space: pre-wrap; ">DISCLAIMER</span><span style="white-space: pre-wrap; ">: this is sharing ideas and brainstorming: I don't know enough of TaksFlow to suggest</span><span style="white-space: pre-wrap; ">): </span></div>
<div><span style="white-space: pre-wrap;">1) break the run loop: add engine.run_async() method which instead of looping the tasks till the flow completes (</span><span style="white-space: pre-wrap; "><a href="http://tinyurl.com/nou2em9">http://tinyurl.com/nou2em9</a>),
only schedules the first batch and returns; </span></div>
<div><a href="https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68">https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68</a></div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Seems like a possiblity, although I'd like to make this still a new engine type that does this (or maybe the existing engine types wrap this change to retain the usage that exists). This just seems like an extension of DELAY thought, where at each iteration
the engine would return DELAYed, then the caller (the one executing) would become aware that this has happened and decide how to handle this accordingly. Exposing 'run_async' doesn't still feel right, maybe its just wording, but it seems like an 'async/lazy'
engine just still has a run() method that returns DELAYed more often than an engine that loops and never returns DELAYed (or only returns DELAYed if a task causes this to happen). To me they are the same run() method, just different implementations of run().
Exposing the semantics of running by naming run_async exposes the underlying execution abstraction, which imho isn't the right way to make this happen.</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><span style="white-space: pre-wrap;">2) introduce engine.task_complete(…) method that marks the current task complete, computes the next tasks ready for execution, schedules them and returns;
</span>loosely, doing this part: <a href="https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85">https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85</a></div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Possibly, although to me this is still exposing the internal of engines to users who shouldn't care (or only care if they are specifying an engine type that gives them access to these details). Allowing public access to these API's worries me in that they
are now the API (which goes back to having an engine type that exposes these, if that’s desired, and if we are willing to accept the consequences of exposing them).</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><span style="white-space: pre-wrap;">3) make executor signal engine back by calling engine.task_complete(…): executor.</span>execute_task() submits the task to the worker queue and calls engine.task_complete(status, results, etc.) instead of waiting for
task completion and returning the results. </div>
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><span style="white-space: pre-wrap;">And looks like it all can be done as a third lazy_engine, without messing the current two engines?</span></div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Possibly, I'm still of the thought that not all engine types should expose this information (due to above public API problem, once its public its public for a long time).</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><span style="white-space: pre-wrap;"></span></div>
<div><br>
</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<ol start="7">
<li>On delay task finishing some API/webhook/other (the mechanism imho shouldn't be tied to webhooks, at least not in taskflow, but should be left up to the user of taskflow to decide how to accomplish this) will be/must be responsible for resuming the engine
and setting the result for the previous delayed task.</li></ol>
</div>
</blockquote>
Oh no, webhook is the way to expose it to 3rd party system. From the library standpoint it's just an API call. </div>
</div>
</div>
</blockquote>
</span><span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div>One can do it even now by getting the appropriate Flow_details, instantiating and engine (flow, flow_details) and running it to continue from where it left out. Is it how you mean it? But I keep on dreaming of a passive version of TaskFlow engine which
treats all tasks the same and exposes one extra method - handle_tasks. </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>I was thinking that, although I'm unsure on the one extra method idea, can u explain more :) What would handle_tasks do? Restart/resume the engine (basically the work u stated 'getting the appropriate Flow_details, instantiating and engine (flow, flow_details)
and running it to continue')? Seems like a tiny helper function if its really wanted, but maybe I'm misunderstanding. It might be connected into a recent taskflow BP @ <a href="https://blueprints.launchpad.net/taskflow/+spec/generic-flow-conductor">https://blueprints.launchpad.net/taskflow/+spec/generic-flow-conductor</a>?</div>
</div>
</blockquote>
<div><br>
</div>
<div>handle_task ops, I meant, engine.task_complete() - engine's call back to receive the task results. See above. </div>
<div>Again, I don't know enough to suggest TaskFlow solutions. Just brainstorming. </div>
</div>
<div>@Kirill, can you also give his view here? <br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<ol start="8">
<li>Repeat 2 -> 7 until all tasks have executed/failed.</li><li>Profit!</li></ol>
</div>
</blockquote>
</div>
<div>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>This seems like it could be accomplished, although there are race conditions in the #6 (what if multiple delayed requests are received at the same time)? What locking is done to ensure that this doesn't cause conflicts?
</div>
</div>
</blockquote>
Engine must handle concurrent calls of mutation methods - start, stop, handle_action. How - differs depending on engine running in multiple threads or in event loop on queues of calls. </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Agreed, to me this requires some level of locking, likely something that the
<a href="https://review.openstack.org/#/c/71167/">tooz</a> library can provide (or in concept is similar to what the
<a href="https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer">
jobboard</a> concept provides, single 'atomic' engine access to a given workflow, ensuring this with a distributing locking scheme, such as zookeeper).</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Does the POC solve that part (no simultaneous step #5 from below)? </div>
</div>
</blockquote>
Yes although we may want to revisit the current solution. </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Is it using some form of database locking? :(</div>
</div>
</blockquote>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>There was a mention of a watch-dog (ideally to ensure that delayed tasks can't just sit around forever), was that implemented?</div>
</div>
</blockquote>
If _delayed_ tasks and 'normal' tasks are treat alike, this is just a matter of timeout as a generic property on a task. So Mistral didn't have to have it. For the proposal above, a separate treatment is necessary for _delayed_ tasks. </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Who responds to the timeout though? Isn't that process the watchdog then? Likely the triggering of a timeout causes something to react (in both cases).</div>
</div>
</blockquote>
The workflow engine IS this watch-dog (and in Mistral, engine is a single manager for all flow executions, in the prototype we call it engine_manager and I hate this name :)) Engine live in process or as a separate process. And it is passive - executed in a
thread of a callee. E.g., in process is either called on messaging event handler thread, or by web method thread.</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Right, which in mistral is a web-server right (aka, wherever mistral is setup) since the tasks finish by calling a rest-endpoint (or something sitting on MQ?)?</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>[1] <a href="https://wiki.openstack.org/wiki/TaskFlow#Retries">https://wiki.openstack.org/wiki/TaskFlow#Retries</a> (new feature!)</div>
</div>
</blockquote>
This is nice. I would call it a 'repeater': running a sub flow several times with various data for various reasons is reacher then 'retry'. </div>
<div>What about the 'retry policy' on individual task? </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>We went back and forth on names a few times (controller, retry, and a few others I think), haha. Retry is what currently stuck :-P</div>
<div><br>
</div>
<div>Repeater does sound nice also though, damn naming of 'things' ;)</div>
<div><br>
</div>
<div>For retries on individual tasks since taskflow supports flow nesting its not that hard to do this (although it might not be the most elegant).</div>
<div><br>
</div>
<div>>>> flow = LinearFlow("blah blah")</div>
<div>>>> flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask()) <<< where the one task is literally one task, haha.</div>
<div><br>
</div>
<div>A running example @ <a href="http://tinyurl.com/kv8q2t8">http://tinyurl.com/kv8q2t8</a></div>
<div><br>
</div>
<div>Room for improvement obviously (isn't there always), but that would/should work.</div>
</div>
</blockquote>
<br>
That may be good enough: when DSL is translated to flow, and the task demands repetition with timeout, it's ok to do this trick under the hood when compiling a flow. </div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask().add(Timeout(timeout))</div>
</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>Yup, in a way most languages compilers do all these types of tricks under the hood (in much much more complicated manners); as long as we retain 'user intention' (aka don't mess how the code executes) we should be able to do any tricks we want (in fact
most compliers do many many tricks). To me the same kind of tricks start to become possible after we get the basics right (can't do optimizations, aka -O2, if u don't have basics in the first place).</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<div>-Josh</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px;">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, March 27, 2014 at 4:43 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Following up on <span style="white-space: pre-wrap; "><a href="http://tinyurl.com/l8gtmsw">http://tinyurl.com/l8gtmsw</a></span> and <span style="white-space: pre-wrap; "><a href="http://tinyurl.com/n3v9lt8:">http://tinyurl.com/n3v9lt8:</a></span> this explains
how Mistral handles long running delegate tasks. Note that a 'passive' workflow engine can handle both normal tasks and delegates the same way. I'll also put that on ActionDesign wiki, after discussion.
<div><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div><font face="Calibri"><span style="font-size: 13px;"><b>Diagram: </b></span></font></div>
<div>
<div style="margin: 0px; "><a href="https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing" style="font-size: 13px;"><font face="Calibri">https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing</font></a></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">1. On start(workflow), engine creates a new workflow execution, computes the first batch of tasks, sends them to ActionRunner [1].</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">2. ActionRunner creates an action and calls action.run(input)</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">3. Action does the work (compute (10!)), produce the results, and return the results to executor. If it returns, status=SUCCESS. If it fails it throws exception, status=ERROR.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">4. ActionRunner notifies Engine that the task is complete task_done(execution, task, status, results)[2]</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">5. Engine computes the next task(s) ready to trigger, according to control flow and data flow, and sends them to ActionRunner.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">6. Like step 2: ActionRunner calls the action's run(input)</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">7. A delegate action doesn't produce results: it calls out the 3rd party system, which is expected to make a callback to a workflow service with the results. It returns to ActionRunner
without results, "immediately". </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">8. ActionRunner marks status=RUNNING [?]</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">9. 3rd party system takes 'long time' == longer then any system component can be assumed to stay alive. </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">10. 3rd party component calls Mistral WebHook which resolves to engine.task_complete(workbook, id, status, results) </span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><b style="font-size: 13px;"><font face="Calibri">Comments: </font></b></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* One Engine handles multiple executions of multiple workflows. It exposes two main operations: start(workflow) and task_complete(execution, task, status, results), and responsible
for defining the next batch of tasks based on control flow and data flow. Engine is passive - it runs in a hosts' thread. Engine and ActionRunner communicate via task queues asynchronously, for details, see
<a href="https://wiki.openstack.org/wiki/Mistral/POC">https://wiki.openstack.org/wiki/Mistral/POC</a> </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Engine doesn't distinct sync and async actions, it doesn't deal with Actions at all. It only reacts to task completions, handling the results, updating the state, and queuing next
set of tasks.</span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Only Action can know and define if it is a delegate or not. Some protocol required to let ActionRunner know that the action is not returning the results immediately. A convention
of returning None may be sufficient. </span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Mistral exposes engine.task_done in the REST API so 3rd party systems can call a web hook.</span></font></div>
</div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">DZ.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; ">
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">[1] I use ActionRunner instead of Executor (current name) to avoid confusion: it is Engine which is responsible for executions, and ActionRunner only runs actions. We should rename
it in the code.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">[2] I use task_done for briefly and out of pure spite, in the code it is conveny_task_results.</span></font></div>
</div>
</div>
</div>
</blockquote>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</span>
</body>
</html>