<div style="font-family: Helvetica; font-size: 13px;"><div>Looks like i'm a bit late to the party, but there is few things i want to add to the discussion before we move it to the IRC. </div><blockquote type="cite"><span><blockquote style="border-left-color: rgb(181, 196, 223); border-left-width: 5px; border-left-style: solid; padding: 0px 0px 0px 5px; margin: 0px 0px 0px 5px;"><div><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><div>For retries on individual tasks since taskflow supports flow nesting its not that hard to do this (although it might not be the most elegant).</div><div><br></div><div>>>> flow = LinearFlow("blah blah")</div><div>>>> flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask()) <<< where the one task is literally one task, haha.</div><div><br></div><div>A running example @ <a href="http://tinyurl.com/kv8q2t8">http://tinyurl.com/kv8q2t8</a></div><div><br></div><div>Room for improvement obviously (isn't there always), but that would/should work.</div></blockquote><div><br></div><div>That may be good enough: when DSL is translated to flow, and the task demands repetition with timeout, it's ok to do this trick under the hood when compiling a flow. </div><div>flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask().add(Timeout(timeout))</div></blockquote><div>Yup, in a way most languages compilers do all these types of tricks under the hood (in much much more complicated manners); as long as we retain 'user intention' (aka don't mess how the code executes) we should be able to do any tricks we want (in fact most compliers do many many tricks). To me the same kind of tricks start to become possible after we get the basics right (can't do optimizations, aka -O2, if u don't have basics in the first place).</div></blockquote><div> </div></div><div>Mistral is not a compiler for TaskFlow, rather an interface. I prefer we don't do that, instead just map Workbook and Flow 1 to 1. If we don't, it would basically means we need additional structure to store Workbooks which should be compiled into Flows, which should be compiled to the execution graph... Isn't that too long? And the only profit here is to keep Retry intact. It's not the kind of integration i was looking for.</div></blockquote></span><div><br></div><div>Mistral is not a compiler for taskflow is an interesting view, although I'd call mistral a user of taskflow and not really a compiler (although mistral may do a lot of translation into taskflow concepts from its own concepts). Maybe there shouldn't be a concept of a workbook then (since taskflow has a similar concept already), or only one of the concepts should exist (not both). I think there is an ongoing blueprint in taskflow that unifies it more around a single concept anyway (see <a href="http://tinyurl.com/mgd8tgk">http://tinyurl.com/mgd8tgk</a>), taskflow has a logbook concept which seems pretty much the same as a workbook (a workbook I think even has the same engine definition as a logbook, haha). To me if one concept isn't good enough then we should consider making that concept more expandable/better/generic so it is good enough (and not inventing 2 of the same concepts instead).</div></blockquote><div>Actually, the idea to make the concept more expandable is exactly my point =) Mistral's Workbook is roughly the same as TaskFlow Flow and have nothing to do with TaskFlow LogBook. The only major difference in case of Workbook is that we have to store it after it was created using API or DSL. I very much like the idea to inherit form basic Flow and add serialization and additional metadata on top of it, but the concept of retrying within a Workbook\Flow is not exactly what we have in mind. Instead of repeating the Flow, we are repeating the Task (which has the same differences between Mistral and TaskFlow as Workbook has) and to use the trick you have proposed we would have to translate the Task into a Flow. How much sense does it make in respect to expandability and generality?</div><div><br></div><div><blockquote type="cite"><span><blockquote style="border-left-color: rgb(181, 196, 223); border-left-width: 5px; border-left-style: solid; padding: 0px 0px 0px 5px; margin: 0px 0px 0px 5px;">Lazy engine should be async and atomic, it should not have its own state, instead it should rely on on some kind of global state (db or in-memory, depending on a type of application). I should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them to execution (either put them in queue or spawn the threads... or send a pidgin, i insist =). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one.</blockquote></span><div><br></div><div>Pidgin execution ftw, haha. In all honesty I'm not yet sure about a task_complete method, this doesn't seem like it should be part of an engine but something outside of it (in a way marking a task complete is marking it in a logbook/workbook that it has finished, which is not really connected to actual execution but can be done by other entities as well). I think a run() (or in taskflow concept execute()) method would be better named as execute_iterations() - which can execute one iteration, yield, execute another iteration, yield (and repeat). If mistral wants to break out of that iteration, it can use the 'break' keyword (while other derived engines can just not break). That seems more pythonic then the above, although lets maybe talk on IRC about this and how it would work.</div></blockquote><div>Ok, it might be a terminology issue once again. Lets tear it apart a bit. There is a concept of Execution in Mistral (which is roughly the same with FlowDetails) which basically represents an instance of the particular Flow we are going to execute with the state and results connected to the every node in the graph. Both Mi and TF engines operates over that Execution, one way or another, and two most frequently used methods of his is 'Set the state and save the results of a particular task' and 'Find the tasks to execute'. In case of Mi engine, this methods are on the surface, in TF engine they are hidden a bit deeper, but they still here. Therefore, i agree this methods may not be part of the engine itself, instead being part of Execution or ExecutionManager (sorry, DZ), but they are definitely the part of the flow execution.</div></div></div>
<div><div><br></div><div><blockquote type="cite"><span><blockquote style="border-left-color: rgb(181, 196, 223); border-left-width: 5px; border-left-style: solid; padding: 0px 0px 0px 5px; margin: 0px 0px 0px 5px;">What i see is bothering Joshua is that worker that handles long running task may just die in a process and there is no way for us to know for sure is it still working or not. It is more important for sync engine because without that it would just hang forever (though it is not clear do sync engine needs long running tasks at all?). In case of sync engine, the role of watch-dog may be performed by the loop, while in case of Mistral it might be a separate process (though i bet there should be something for resending the message in the RabbitMQ itself). The common solution is not necessary here.</blockquote></span><div><br></div><div>Ya, I think u are hitting this issue, if a worker acks a message saying its 'working on it' then the worker dies, this is where a message queue ack model won't really help in solving, the engine will believe that the worker is working on it (since hey the worker acked it) but the engine will really have no idea that the worker is actually dead. Of course timeouts can be used to get around this but imho that’s really un-elegant when something like zookeeper (or similar) can be used to be notified of this worker death <i>immediately</i>; which means no timeouts are needed at all, and this separate process or engine can be notified of this death (and resolve it immediately). I don't believe this kind of '<i>liveness</i>' connection is possible with rabbit (we also must think of other things besides rabbit, like qpid and such) but I might be wrong, I thought once a worker acks a message then whoever receives that ack will believe the work has started and will be finished someday in the future (aka, no connection that the work is actually in progress). Anyways, we can discuss this more since I think its a common confusion point :-)</div></blockquote><div>As far as i understand, what Renat is proposing here is not to acknowledge the message until it was successfully executed. I'm not sure how exactly RabbitMQ will react in that case, though you are right in the idea that we must investigate the other solutions more carefully.</div></div><div><br></div><div>-- </div><div>Kirill Izotov<br></div><div><br></div></div>
<p style="color: #A0A0A8;">четверг, 3 апреля 2014 г. в 1:51, Joshua Harlow написал:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1251">
<div>And I'll answer to Kirill, lol.</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Kirill Izotov <<a href="mailto:enykeev@stackstorm.com">enykeev@stackstorm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, April 1, 2014 at 11:38 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>Ok, i'll try to answer to both of you. </div>
<div>
<div><br>
</div>
<div><blockquote type="cite"><blockquote type="cite"><div>
</div></blockquote><div><blockquote type="cite"><div><blockquote type="cite"><div>
<div>For retries on individual tasks since taskflow supports flow nesting its not that hard to do this (although it might not be the most elegant).</div>
<div><br>
</div>
<div>>>> flow = LinearFlow("blah blah")</div>
<div>>>> flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask()) <<< where the one task is literally one task, haha.</div>
<div><br>
</div>
<div>A running example @ <a href="http://tinyurl.com/kv8q2t8">http://tinyurl.com/kv8q2t8</a></div>
<div><br>
</div>
<div>Room for improvement obviously (isn't there always), but that would/should work.</div>
</div></blockquote><div><br>
</div>
<div>That may be good enough: when DSL is translated to flow, and the task demands repetition with timeout, it's ok to do this trick under the hood when compiling a flow. </div>
<div>flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask().add(Timeout(timeout))</div>
</div></blockquote><div>Yup, in a way most languages compilers do all these types of tricks under the hood (in much much more complicated manners); as long as we retain 'user intention' (aka don't mess how the code executes) we should be able to do any tricks we want (in fact
most compliers do many many tricks). To me the same kind of tricks start to become possible after we get the basics right (can't do optimizations, aka -O2, if u don't have basics in the first place).</div>
</div></blockquote>
<div> </div>
</div>
<div>Mistral is not a compiler for TaskFlow, rather an interface. I prefer we don't do that, instead just map Workbook and Flow 1 to 1. If we don't, it would basically means we need additional structure to store Workbooks which should be compiled into Flows,
which should be compiled to the execution graph... Isn't that too long? And the only profit here is to keep Retry intact. It's not the kind of integration i was looking for.</div>
</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Mistral is not a compiler for taskflow is an interesting view, although I'd call mistral a user of taskflow and not really a compiler (although mistral may do a lot of translation into taskflow concepts from its own concepts). Maybe there shouldn't be
a concept of a workbook then (since taskflow has a similar concept already), or only one of the concepts should exist (not both). I think there is an ongoing blueprint in taskflow that unifies it more around a single concept anyway (see <a href="http://tinyurl.com/mgd8tgk">http://tinyurl.com/mgd8tgk</a>),
taskflow has a logbook concept which seems pretty much the same as a workbook (a workbook I think even has the same engine definition as a logbook, haha). To me if one concept isn't good enough then we should consider making that concept more expandable/better/generic
so it is good enough (and not inventing 2 of the same concepts instead).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>
<div><br>
</div>
<div>
<blockquote type="cite"><div>
</div><blockquote type="cite"><div>
</div></blockquote></blockquote></div><blockquote type="cite"><div>
<div><blockquote type="cite"><div><blockquote type="cite"><div>
<div>Not all ways of using taskflow I think want to go 'asleep', some active conductors I would think actually stay 'awake', that’s why I was thinking this would be a comprimise, the usage of 'DELAY' (via exception or return) would allow a task to say it is
going to be finished sometime in the future (btw started this @ <a href="https://review.openstack.org/#/c/84277/">
https://review.openstack.org/#/c/84277/</a> --- WIP, likely won't go into taskflow 0.2 but seems like a reasonable start on something like this proposal). To me offering just the going asleep way means that all users of taskflow need to have some type of entrypoint
(rest, mq, other…) that restarts the engine on result finishing. I'd rather not force that model onto all users (since that seems inappropriate).</div>
</div></blockquote>
I understand and agree with not enforcing this model on all users. On a flip side, TaskFlow enforces the current model on Mistral :) It calls for supporting more models (to your earlier point of 3rd mode of operations). The BP proposal is a step in the direction
but still a compromise. How can we support both 'active' and 'passive' models, sharing the same 'library' - of all things responsible for control flow and data flow?<br>
</div></blockquote><font color="#002162"><br>
</font>
<div>To me this is what engines 'types' are for, to support different execution models while using as many of the same components as possible (making usage as seamless as can be).</div>
</div>
<div><br>
</div>
<div>
<blockquote type="cite"><div>
<div>Here is a strawman (DISCLAIMER: this is sharing ideas and brainstorming: I don't know enough of TaksFlow to suggest): </div>
<div>1) break the run loop: add engine.run_async() method which instead of looping the tasks till the flow completes (<a href="http://tinyurl.com/nou2em9">http://tinyurl.com/nou2em9</a>),</div>
<div> only schedules the first batch and returns; </div>
<div><a href="https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68">https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68</a></div>
</div></blockquote>
<div><br>
</div>
<div>Seems like a possiblity, although I'd like to make this still a new engine type that does this (or maybe the existing engine types wrap this change to retain the usage that exists). This just seems like an extension of DELAY thought, where at each iteration
the engine would return DELAYed, then the caller (the one executing) would become aware that this has happened and decide how to handle this accordingly. Exposing 'run_async' doesn't still feel right, maybe its just wording, but it seems like an 'async/lazy'
engine just still has a run() method that returns DELAYed more often than an engine that loops and never returns DELAYed (or only returns DELAYed if a task causes this to happen). To me they are the same run() method, just different implementations of run().
Exposing the semantics of running by naming run_async exposes the underlying execution abstraction, which imho isn't the right way to make this happen.</div>
</div>
</div></blockquote><div>
</div>
<div>I agree that we probably need new engine for that kind of changes and, as Renat already said in another thread, lazy model seems to be more basic and it would be easier to build sync engine on top of that rather than other way around. Yes, it will entail
a lot of changes in engines that are currently here, but it seems like the only way to get something that would fit us both.</div>
<div><br>
</div>
<div>Since it seems like we are getting some kind of agreement here, we should probably start shifting to the plane where we discuss the design of the new engine, rather than the need of one. The idea has been spread over to many places, so i'll try to gather
it back. </div>
</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Lets see if we can do this maybe in IRC, design over email I don't think will work out so well cause it will just imho lead to a lot of back and forth and confusion that can just be avoided if we talk on IRC (where those kind of confusions can just be
resolved immediately).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>
<div><br>
</div>
<div>Lazy engine should be async and atomic, it should not have its own state, instead it should rely on on some kind of global state (db or in-memory, depending on a type of application). I should have at least two methods: run and task_complete. Run method
should calculate the first batch of tasks and schedule them to execution (either put them in queue or spawn the threads... or send a pidgin, i insist =). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that
became available due to resolution of this one.</div>
</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Pidgin execution ftw, haha. In all honesty I'm not yet sure about a task_complete method, this doesn't seem like it should be part of an engine but something outside of it (in a way marking a task complete is marking it in a logbook/workbook that it has
finished, which is not really connected to actual execution but can be done by other entities as well). I think a run() (or in taskflow concept execute()) method would be better named as execute_iterations() - which can execute one iteration, yield, execute
another iteration, yield (and repeat). If mistral wants to break out of that iteration, it can use the 'break' keyword (while other derived engines can just not break). That seems more pythonic then the above, although lets maybe talk on IRC about this and
how it would work.</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>
<div><br>
</div>
<div>Then, on top of it you can build sync engine by introducing Futures. You are using async.run() to schedule the tasks by transforming them to Futures and then starting a loop, checking Futures for completion and sending their results to async.task_complete()
which would produce even more Futures to check over. Just the same way TaskFlow do it right now.</div>
<div><br>
</div>
<div>On the Mistral side we are using Lazy engine by patching async.run to the API (or engine queue) and async.task_complete to the worker queue result channel (and the API for long running tasks). We still sharing the same graph_analyzer, but instead of relying
on loop and Futures, we are handling execution in a scalable and robust way.</div>
<div><br>
</div>
<div>The reason i'm proposing to extract Futures from async engine is because they won't work if we have multiple engines that should handle the task results concurrently (and without that there will be no scalability).</div>
<div><br>
</div>
<div>What i see is bothering Joshua is that worker that handles long running task may just die in a process and there is no way for us to know for sure is it still working or not. It is more important for sync engine because without that it would just hang
forever (though it is not clear do sync engine needs long running tasks at all?). In case of sync engine, the role of watch-dog may be performed by the loop, while in case of Mistral it might be a separate process (though i bet there should be something for
resending the message in the RabbitMQ itself). The common solution is not necessary here.</div>
</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Ya, I think u are hitting this issue, if a worker acks a message saying its 'working on it' then the worker dies, this is where a message queue ack model won't really help in solving, the engine will believe that the worker is working on it (since hey
the worker acked it) but the engine will really have no idea that the worker is actually dead. Of course timeouts can be used to get around this but imho that’s really un-elegant when something like zookeeper (or similar) can be used to be notified of this
worker death <i>immediately</i>; which means no timeouts are needed at all, and this separate process or engine can be notified of this death (and resolve it immediately). I don't believe this kind of '<i>liveness</i>' connection is possible with rabbit (we
also must think of other things besides rabbit, like qpid and such) but I might be wrong, I thought once a worker acks a message then whoever receives that ack will believe the work has started and will be finished someday in the future (aka, no connection
that the work is actually in progress). Anyways, we can discuss this more since I think its a common confusion point :-)</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>
<div><br>
</div>
<div>What am i missing here?</div>
<div><br>
</div>
<div>-- </div>
<div>Kirill Izotov<br>
</div>
<div><br>
</div>
</div>
<p style="color: #A0A0A8;">ñðåäà, 2 àïðåëÿ 2014 ã. â 5:45, Joshua Harlow íàïèñàë:</p><blockquote type="cite"><div>
<span>
<div>
<div>
<div>Cool, my thoughts added ;)</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, April 1, 2014 at 2:51 PM<br>
<span style="font-weight:bold">To: </span>Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Cc: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
Even more responses inline :)</div>
<div>
<div>On Mar 31, 2014, at 8:44 PM, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div>
<br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div>More responses inline :)</div>
<div><br>
</div>
<span>
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Date: </span>Monday, March 31, 2014 at 5:59 PM<br>
<span style="font-weight:bold">To: </span>Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Cc: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Inline...<br>
<div>
<div>On Mar 27, 2014, at 5:10 PM, Joshua Harlow <<a href="mailto:harlowja@yahoo-inc.com">harlowja@yahoo-inc.com</a>> wrote:</div>
<br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">Thanks for the description!</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">The steps here seem very much like what a taskflow engine does (which is good).</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; "><br>
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; ">To connect this to how I think could work in taskflow.</div>
<ol>
<li><font face="Calibri,sans-serif">Someone creates tasks/flows describing the work
<i>to-be-done</i> (converting a DSL -> taskflow tasks/flows/retry[1] objects</font><font face="Calibri,sans-serif">…)</font></li><li><font face="Calibri,sans-serif">On execute(workflow) </font><span style="font-family: Calibri; ">engine creates a new workflow execution, computes the first batch of tasks, creates executor for those tasks (remote, local</span>…) and executes those tasks.</li><li>Waits for response back from <a href="http://docs.python.org/dev/library/concurrent.futures.html">
futures</a> returned from executor.</li><li>Receives futures responses (or receives new response <i>DELAY</i>, for example), or exceptions…</li><li>Continues sending out batches of tasks that can be still be executing (aka tasks that don't have dependency on output of delayed tasks).</li><li>If any delayed tasks after repeating #2-5 as many times as it can, the engine will shut itself down (see
<a href="http://tinyurl.com/l3x3rrb">http://tinyurl.com/l3x3rrb</a>).</li></ol>
</div>
</div>
</div></blockquote>
Why would engine treat long running tasks differently? The model Mistral tried out is the engine sends the batch of tasks and goes asleep; the 'worker/executor' is calling engine back when the task(s) complete. Can it be applied </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>Not all ways of using taskflow I think want to go 'asleep', some active conductors I would think actually stay 'awake', that’s why I was thinking this would be a comprimise, the usage of 'DELAY' (via exception or return) would allow a task to say it is
going to be finished sometime in the future (btw started this @ <a href="https://review.openstack.org/#/c/84277">https://review.openstack.org/#/c/84277</a>/ --- WIP, likely won't go into taskflow 0.2 but seems like a reasonable start on something like this
proposal). To me offering just the going asleep way means that all users of taskflow need to have some type of entrypoint (rest, mq, other…) that restarts the engine on result finishing. I'd rather not force that model onto all users (since that seems inappropriate).</div>
</div>
</div>
</div></blockquote>
I understand and agree with not enforcing this model on all users. On a flip side, TaskFlow enforces the current model on Mistral :) It calls for supporting more models (to your earlier point of 3rd mode of operations). The BP proposal is a step in the direction
but still a compromise. How can we support both 'active' and 'passive' models, sharing the same 'library' - of all things responsible for control flow and data flow?</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>To me this is what engines 'types' are for, to support different execution models while using as many of the same components as possible (making usage as seamless as can be).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><span style="white-space: pre-wrap;">Here is a strawman (</span><span style="white-space: pre-wrap; ">DISCLAIMER</span><span style="white-space: pre-wrap; ">: this is sharing ideas and brainstorming: I don't know enough of TaksFlow to suggest</span><span style="white-space: pre-wrap; ">): </span></div>
<div><span style="white-space: pre-wrap;">1) break the run loop: add engine.run_async() method which instead of looping the tasks till the flow completes (</span><span style="white-space: pre-wrap; "><a href="http://tinyurl.com/nou2em9">http://tinyurl.com/nou2em9</a>),
only schedules the first batch and returns; </span></div>
<div><a href="https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68">https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68</a></div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Seems like a possiblity, although I'd like to make this still a new engine type that does this (or maybe the existing engine types wrap this change to retain the usage that exists). This just seems like an extension of DELAY thought, where at each iteration
the engine would return DELAYed, then the caller (the one executing) would become aware that this has happened and decide how to handle this accordingly. Exposing 'run_async' doesn't still feel right, maybe its just wording, but it seems like an 'async/lazy'
engine just still has a run() method that returns DELAYed more often than an engine that loops and never returns DELAYed (or only returns DELAYed if a task causes this to happen). To me they are the same run() method, just different implementations of run().
Exposing the semantics of running by naming run_async exposes the underlying execution abstraction, which imho isn't the right way to make this happen.</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><span style="white-space: pre-wrap;">2) introduce engine.task_complete(…) method that marks the current task complete, computes the next tasks ready for execution, schedules them and returns;
</span>loosely, doing this part: <a href="https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85">https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85</a></div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Possibly, although to me this is still exposing the internal of engines to users who shouldn't care (or only care if they are specifying an engine type that gives them access to these details). Allowing public access to these API's worries me in that they
are now the API (which goes back to having an engine type that exposes these, if that’s desired, and if we are willing to accept the consequences of exposing them).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><span style="white-space: pre-wrap;">3) make executor signal engine back by calling engine.task_complete(…): executor.</span>execute_task() submits the task to the worker queue and calls engine.task_complete(status, results, etc.) instead of waiting for
task completion and returning the results. </div>
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><span style="white-space: pre-wrap;">And looks like it all can be done as a third lazy_engine, without messing the current two engines?</span></div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Possibly, I'm still of the thought that not all engine types should expose this information (due to above public API problem, once its public its public for a long time).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><span style="white-space: pre-wrap;"></span></div>
<div><br>
</div>
<div><br>
</div>
<div><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<ol start="7">
<li>On delay task finishing some API/webhook/other (the mechanism imho shouldn't be tied to webhooks, at least not in taskflow, but should be left up to the user of taskflow to decide how to accomplish this) will be/must be responsible for resuming the engine
and setting the result for the previous delayed task.</li></ol>
</div>
</div>
</div></blockquote>
Oh no, webhook is the way to expose it to 3rd party system. From the library standpoint it's just an API call. </div>
</div>
</div>
</div>
</div></blockquote></span><span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div>One can do it even now by getting the appropriate Flow_details, instantiating and engine (flow, flow_details) and running it to continue from where it left out. Is it how you mean it? But I keep on dreaming of a passive version of TaskFlow engine which
treats all tasks the same and exposes one extra method - handle_tasks. </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>I was thinking that, although I'm unsure on the one extra method idea, can u explain more :) What would handle_tasks do? Restart/resume the engine (basically the work u stated 'getting the appropriate Flow_details, instantiating and engine (flow, flow_details)
and running it to continue')? Seems like a tiny helper function if its really wanted, but maybe I'm misunderstanding. It might be connected into a recent taskflow BP @ <a href="https://blueprints.launchpad.net/taskflow/+spec/generic-flow-conductor">https://blueprints.launchpad.net/taskflow/+spec/generic-flow-conductor</a>?</div>
</div>
</div>
</div></blockquote><div><br>
</div>
<div>handle_task ops, I meant, engine.task_complete() - engine's call back to receive the task results. See above. </div>
<div>Again, I don't know enough to suggest TaskFlow solutions. Just brainstorming. </div>
</div>
<div>@Kirill, can you also give his view here? <br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<ol start="8">
<li>Repeat 2 -> 7 until all tasks have executed/failed.</li><li>Profit!</li></ol>
</div>
</div>
</div></blockquote></div>
<div><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>This seems like it could be accomplished, although there are race conditions in the #6 (what if multiple delayed requests are received at the same time)? What locking is done to ensure that this doesn't cause conflicts?
</div>
</div>
</div>
</div></blockquote>
Engine must handle concurrent calls of mutation methods - start, stop, handle_action. How - differs depending on engine running in multiple threads or in event loop on queues of calls. </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>Agreed, to me this requires some level of locking, likely something that the
<a href="https://review.openstack.org/#/c/71167/">tooz</a> library can provide (or in concept is similar to what the
<a href="https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer">
jobboard</a> concept provides, single 'atomic' engine access to a given workflow, ensuring this with a distributing locking scheme, such as zookeeper).</div>
<div><br>
</div>
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Does the POC solve that part (no simultaneous step #5 from below)? </div>
</div>
</div>
</div></blockquote>
Yes although we may want to revisit the current solution. </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>Is it using some form of database locking? :(</div>
</div>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>There was a mention of a watch-dog (ideally to ensure that delayed tasks can't just sit around forever), was that implemented?</div>
</div>
</div>
</div></blockquote>
If _delayed_ tasks and 'normal' tasks are treat alike, this is just a matter of timeout as a generic property on a task. So Mistral didn't have to have it. For the proposal above, a separate treatment is necessary for _delayed_ tasks. </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>Who responds to the timeout though? Isn't that process the watchdog then? Likely the triggering of a timeout causes something to react (in both cases).</div>
</div>
</div>
</div></blockquote>
The workflow engine IS this watch-dog (and in Mistral, engine is a single manager for all flow executions, in the prototype we call it engine_manager and I hate this name :)) Engine live in process or as a separate process. And it is passive - executed in a
thread of a callee. E.g., in process is either called on messaging event handler thread, or by web method thread.</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Right, which in mistral is a web-server right (aka, wherever mistral is setup) since the tasks finish by calling a rest-endpoint (or something sitting on MQ?)?</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>[1] <a href="https://wiki.openstack.org/wiki/TaskFlow#Retries">https://wiki.openstack.org/wiki/TaskFlow#Retries</a> (new feature!)</div>
</div>
</div>
</div></blockquote>
This is nice. I would call it a 'repeater': running a sub flow several times with various data for various reasons is reacher then 'retry'. </div>
<div>What about the 'retry policy' on individual task? </div>
</div>
</div>
</div>
</div></blockquote></span>
<div><br>
</div>
<div>We went back and forth on names a few times (controller, retry, and a few others I think), haha. Retry is what currently stuck :-P</div>
<div><br>
</div>
<div>Repeater does sound nice also though, damn naming of 'things' ;)</div>
<div><br>
</div>
<div>For retries on individual tasks since taskflow supports flow nesting its not that hard to do this (although it might not be the most elegant).</div>
<div><br>
</div>
<div>>>> flow = LinearFlow("blah blah")</div>
<div>>>> flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask()) <<< where the one task is literally one task, haha.</div>
<div><br>
</div>
<div>A running example @ <a href="http://tinyurl.com/kv8q2t8">http://tinyurl.com/kv8q2t8</a></div>
<div><br>
</div>
<div>Room for improvement obviously (isn't there always), but that would/should work.</div>
</div>
</div>
</div></blockquote><br>
That may be good enough: when DSL is translated to flow, and the task demands repetition with timeout, it's ok to do this trick under the hood when compiling a flow. </div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
flow.add(LinearFlow("subblahblah", retry=XYZ).add(OneTask().add(Timeout(timeout))</div>
</div>
</div>
</div>
</blockquote></span>
<div><br>
</div>
<div>Yup, in a way most languages compilers do all these types of tricks under the hood (in much much more complicated manners); as long as we retain 'user intention' (aka don't mess how the code executes) we should be able to do any tricks we want (in fact
most compliers do many many tricks). To me the same kind of tricks start to become possible after we get the basics right (can't do optimizations, aka -O2, if u don't have basics in the first place).</div>
<div><br>
</div>
<span><blockquote style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<div>-Josh</div>
<div><br>
</div>
<span><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br><blockquote type="cite"><div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<span style="font-family: Calibri, sans-serif; font-size: 14px;">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>Dmitri Zimine <<a href="mailto:dz@stackstorm.com">dz@stackstorm.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, March 27, 2014 at 4:43 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[openstack-dev] [Mistral] How Mistral handling long running delegate tasks<br>
</div>
<div><br>
</div><blockquote type="cite"><div>
<div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Following up on <span style="white-space: pre-wrap; "><a href="http://tinyurl.com/l8gtmsw">http://tinyurl.com/l8gtmsw</a></span> and <span style="white-space: pre-wrap; "><a href="http://tinyurl.com/n3v9lt8:">http://tinyurl.com/n3v9lt8:</a></span> this explains
how Mistral handles long running delegate tasks. Note that a 'passive' workflow engine can handle both normal tasks and delegates the same way. I'll also put that on ActionDesign wiki, after discussion.
<div><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div><font face="Calibri"><span style="font-size: 13px;"><b>Diagram: </b></span></font></div>
<div>
<div style="margin: 0px; "><a href="https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing" style="font-size: 13px;"><font face="Calibri">https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing</font></a></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">1. On start(workflow), engine creates a new workflow execution, computes the first batch of tasks, sends them to ActionRunner [1].</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">2. ActionRunner creates an action and calls action.run(input)</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">3. Action does the work (compute (10!)), produce the results, and return the results to executor. If it returns, status=SUCCESS. If it fails it throws exception, status=ERROR.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">4. ActionRunner notifies Engine that the task is complete task_done(execution, task, status, results)[2]</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">5. Engine computes the next task(s) ready to trigger, according to control flow and data flow, and sends them to ActionRunner.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">6. Like step 2: ActionRunner calls the action's run(input)</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">7. A delegate action doesn't produce results: it calls out the 3rd party system, which is expected to make a callback to a workflow service with the results. It returns to ActionRunner
without results, "immediately". </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">8. ActionRunner marks status=RUNNING [?]</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">9. 3rd party system takes 'long time' == longer then any system component can be assumed to stay alive. </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">10. 3rd party component calls Mistral WebHook which resolves to engine.task_complete(workbook, id, status, results) </span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><b style="font-size: 13px;"><font face="Calibri">Comments: </font></b></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* One Engine handles multiple executions of multiple workflows. It exposes two main operations: start(workflow) and task_complete(execution, task, status, results), and responsible
for defining the next batch of tasks based on control flow and data flow. Engine is passive - it runs in a hosts' thread. Engine and ActionRunner communicate via task queues asynchronously, for details, see
<a href="https://wiki.openstack.org/wiki/Mistral/POC">https://wiki.openstack.org/wiki/Mistral/POC</a> </span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Engine doesn't distinct sync and async actions, it doesn't deal with Actions at all. It only reacts to task completions, handling the results, updating the state, and queuing next
set of tasks.</span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Only Action can know and define if it is a delegate or not. Some protocol required to let ActionRunner know that the action is not returning the results immediately. A convention
of returning None may be sufficient. </span></font></div>
<div style="margin: 0px; min-height: 22px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">* Mistral exposes engine.task_done in the REST API so 3rd party systems can call a web hook.</span></font></div>
</div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">DZ.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; ">
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">[1] I use ActionRunner instead of Executor (current name) to avoid confusion: it is Engine which is responsible for executions, and ActionRunner only runs actions. We should rename
it in the code.</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;"><br>
</span></font></div>
<div style="margin: 0px; "><font face="Calibri"><span style="font-size: 13px;">[2] I use task_done for briefly and out of pure spite, in the code it is conveny_task_results.</span></font></div>
</div>
</div>
</div>
</div>
</div></blockquote></span></div>
</div>
</div></blockquote></div>
<br>
</div>
</div>
</div>
</div></blockquote></span></div>
</div>
</div></blockquote></div>
<br>
</div>
</div>
</blockquote></span></div>
<div>
<div>_______________________________________________</div>
<div>OpenStack-dev mailing list</div>
<div><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a></div>
<div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
</div>
</div>
</span></div></blockquote><div><br>
</div>
</div>
</div>
</blockquote></span>
</div></div></span>
</blockquote>
<div>
<br>
</div>