<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div apple-content-edited="true"><div>On 02 Apr 2014, at 06:00, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div></div><div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;">
<div>More inline.</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, April 1, 2014 at 2:59 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral][TaskFlow] Long running actions<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br>
<div>
<div>On Apr 1, 2014, at 3:43 AM, Renat Akhmerov <<a href="mailto:rakhmerov@mirantis.com">rakhmerov@mirantis.com</a>> wrote:</div>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div apple-content-edited="true">
<div>On 25 Mar 2014, at 01:51, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div>
</div>
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>The first execution model I would call the local execution model, this model involves forming tasks and flows and then executing them inside an application, that application is running for the duration of the workflow (although if it crashes it can re-establish
the task and flows that it was doing and attempt to resume them). This could also be what openstack projects would call the 'conductor' approach where nova, ironic, trove have a conductor which manages these long-running actions (the conductor is alive/running
throughout the duration of these workflows, although it may be restarted while running). The restarting + resuming part is something that openstack hasn't handled so gracefully currently, typically requiring either some type of cleanup at restart (or by operations),
with taskflow using this model the resumption part makes it possible to resume from the last saved state (this connects into the persistence model that taskflow uses, the state transitions, how execution occurrs itself...). </div>
<div><br>
</div>
<div>The second execution model is an extension of the first, whereby there is still a type of 'conductor' that is managing the life-time of the workflow, but instead of locally executing tasks in the conductor itself tasks are now executed on remote-workers
(see <a href="http://tinyurl.com/lf3yqe4">http://tinyurl.com/lf3yqe4</a></div>
<div>). The engine currently still is 'alive' for the life-time of the execution, although the work that it is doing is relatively minimal (since its not actually executing any task code, but proxying those requests to others works). The engine while running
does the conducting of the remote-workers (saving persistence details, doing state-transtions, getting results, sending requests to workers…).</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>These two execution models are special cases of what you call “lazy execution model” (or passive as we call it). To illustrate this idea we can take a look at the first sequence diagram at [0], we basically will see the following interaction:</div>
<div><br>
</div>
<div>1) engine --(task)--> queue --(task)--> worker</div>
<div>2) execute task</div>
<div>3) worker --(result)--> queue --(result)--> engine</div>
<div><br>
</div>
<div>This is how TaskFlow worker based model works.</div>
<div><br>
</div>
<div>If we loosen the requirement in 3) and assume that not only worker can send a task result back to engine we’ll get our passive model. Instead of worker it can be anything else (some external system) that knows how to make this call. A particular way is
not too important, it can be a direct message or it can be hidden behind an API method. In Mistral it’s now a REST API method however we’re about to decouple engine from REST API so that engine is a standalone process and listens to a queue. So worker-based
model is basically the same with the only strict requirement that only worker sends a result back.</div>
<div><br>
</div>
<div>In order to implement local execution model on top of “lazy execution model” we just need to abstract a transport (queue) so that we can use an in-process transport. That’s it. It’s what Mistral already has implemented. Again, we see that “lazy execution
model” is more universal.</div>
<div><br>
</div>
<div>IMO this “lazy execution model” should be the main execution model that TaskFlow supports, others can be easily implemented on top of it. But the opposite assertion is wrong. IMO this is the most important obstacle in all our discussions, the reason why
we don’t always understand each other well enough. I know it may be a lot of work to shift a paradigm in TaskFlow team but if we did that we would get enough freedom for using TaskFlow in lots of cases.</div>
<div><br>
</div>
<div>Let me know what you think. I might have missed something.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
DZ: Interesting idea! So that other models of execution are based on lazy execution model? TaskFlow implements this, we can use it, and for other clients more convenient higher level execution models are provided? Interesting. Makes sense.</div>
<div>@Joshua? @Kirill? Others? </div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
<div>I think this is likely possible, which is simiar to whats in <a href="http://tinyurl.com/k3s2gmy">
http://tinyurl.com/k3s2gmy</a>, engine types can be built from each other (and if we wanted to alter the structure that exists in taskflow) then sure. But see that message for more of my concerns around exposing that engine API to library users (I think it
could have its usage in mistral to expose this, but I'm not sure its useful for elsewhere, and once its public engine API, its public for a very long time).</div>
</div></blockquote><br></div><div>What are we waiting for? Let’s code it up! :)</div><div><br></div><div>Renat Akhmerov</div><div>@ Mirantis Inc.</div><div><br></div></body></html>