<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Thanks for the write-up krill.</div>
<div><br>
</div>
<div>Also some adjustments,</div>
<div><br>
</div>
<div>Both points are good, and putting some of this on @ <a href="https://etherpad.openstack.org/p/taskflow-mistral-details">https://etherpad.openstack.org/p/taskflow-mistral-details</a> so that we can have it actively noted (feel free to adjust it).</div>
<div><br>
</div>
<div>I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that.</div>
<div><br>
</div>
<div>For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas
I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow & mistral & openstack can be the best it can be (pigeons not included).</div>
<div><br>
</div>
<div>Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks!</div>
<div><br>
</div>
<div>[1] <a href="https://wiki.openstack.org/wiki/TaskFlow#Retries">https://wiki.openstack.org/wiki/TaskFlow#Retries</a></div>
<div>[2] <a href="https://review.openstack.org/#/c/86470">https://review.openstack.org/#/c/86470</a></div>
<div>[3] <a href="https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z">https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z</a> </div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Kirill Izotov <<a href="mailto:enykeev@stackstorm.com">enykeev@stackstorm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 10, 2014 at 9:20 PM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>" <<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>
<div class="">
<div class="">Hi everyone,</div>
<div class=""><br>
</div>
<div class="">This is a summary to the prototype integration we did not too long ago: <a href="http://github.com/enykeev/mistral/pull/1">http://github.com/enykeev/mistral/pull/1</a>. Hope it would shed some light on the aspects of the integration we are struggling
with.</div>
<div class=""><br>
</div>
<div class="">There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change.</div>
<div class=""><br>
</div>
<div class="">There are two main sides of the story. One is engine. The other is flow control capabilities. </div>
<div class=""><br>
</div>
<div class="">1) THE ENGINE </div>
<div class="">The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and
it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts.</div>
<div class=""><br>
</div>
<div class="">Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run
method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to
resolution of this one.</div>
<div class=""><br>
</div>
<div class="">The desired use of lazy engine in Mistral is illustrated here: <a href="https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture">
https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture</a>. It should support long running tasks and survive engine process restart without loosing the state of the running actions. So it must be passive (lazy) and persistent. </div>
<div class=""><br>
</div>
<div class="">On Mistral side we are using Lazy engine by patching async.run directly to the API (or engine queue) and async.task_complete to the worker queue result channel (and the API for long running tasks). We are still sharing the same graph_analyzer,
but instead of relying on loop and Futures, we are handling the execution ourselves in a scalable and robust way.</div>
<div class=""><br>
</div>
<div class="">Then, on top of it you can build a sync engine by introducing Futures. You are using async.run() to schedule tasks by transforming them to Futures and then starting a loop, checking Futures for completion and sending their results to async.task_complete()
which would produce even more Futures to check over. Just the same way TaskFlow do it right now.</div>
<div class=""><br>
</div>
<div class="">The reason I'm proposing to extract Futures from async engine is because they won't work if we have multiple engine processes that should handle the task results concurrently (and without that there will be no scalability).</div>
<div class=""><br>
</div>
<div class="">2) THE FLOW CONTROL CAPABILITIES</div>
<div class=""><br>
</div>
<div class="">Since we treat TaskFlow as a library we expect them to provide us with a number of primitives to build our workflow with them. Most important of them to us for the moment are Direct Transitions, and Conditional Transitions. </div>
<div class=""><br>
</div>
<div class="">The current implementation of flow transitions in TaskFlow are built on top of data flow dependencies where each task provides some data to the flow and requires some data to be present prior being executed. In other words, you are starting to
build your flow tree from the last task through the first one by adding their requirements to the tree. All the tasks of successfully finished flow should be successfully finished too. If one of the tasks finishes with error, the whole flow will be reverted
back to its initial state unconditionally.</div>
<div class=""><br>
</div>
<div class="">At the same time, Mistral use cases require direct control on the order of the task execution, with top-to-bottom scheme where the next task will be determined based on the results of the execution of the current one. This way to successfully
finish a flow you don't have to execute all tasks in it. Besides, the error in execution of a particular task may cause execution of another one. The workflow examples (in pseudo DSL) are here:
<a href="https://github.com/dzimine/mistral-workflows/tree/add-usecases">https://github.com/dzimine/mistral-workflows/tree/add-usecases</a></div>
<div class=""><br>
</div>
<div class="">There is also a handful of small issues, but these two differences cover most basic parts of TaskFlow thus block us from integration and require substantial changes in TaskFlow engine design. Inability to make such changes will directly result
in Mistral not being able to meet its requirements and thus rendering the whole project useless for its users.</div>
<div><br>
</div>
</div>
</div>
<div>
<div>-- </div>
<div>Kirill Izotov<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</span>
</body>
</html>