<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks Winson,<div class=""><br class=""></div><div class="">Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job!</div><div class=""><br class=""></div><div class="">However, all additional comments and questions are more than welcomed!</div><div class=""><br class=""></div><div class=""><br class=""><div class="">
<div class="">Renat Akhmerov</div><div class="">@ Mirantis Inc.</div><div class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On 24 Dec 2014, at 04:32, W Chan <<a href="mailto:m4d.coder@gmail.com" class="">m4d.coder@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints.</div><div class="">* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment" class="">https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment</a><br class=""></div><div class="">* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-global-context" class="">https://blueprints.launchpad.net/mistral/+spec/mistral-global-context</a></div><div class="">* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values" class="">https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values</a></div><div class="">* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context" class="">https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context</a></div></div><div class=""><br class=""></div><div class="">Please refer to the following threads for backgrounds.</div><div class=""><div class="">* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html" class="">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html</a></div><div class="">* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html" class="">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html</a><br class=""></div></div><div class="">* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html" class="">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html</a><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><b class="">Workflow Context Scope</b></div><div class="">1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs<br class=""></div><div class="">2. context are passed by value (copy.deepcopy) to children</div><div class="">3. change to context is passed to parent only when it's explicitly published at the end of the child execution</div><div class="">4. change to context at the parent (after a publish from a child) is passed to subsequent children</div></div><div class="" style="font-family: Helvetica; font-size: 12px;"><br class=""></div><span class="" style="font-family: Helvetica; font-size: 12px;"><b class="">Environment Variables</b></span></div><div class=""><font face="Helvetica" class="">Solves the problem for quickly passing pre-defined inputs to a WF execution. In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc. </font><span class="" style="font-family: Helvetica; font-size: 12px;">We should implement an API and DB model where users can pre-defined different environments with their own set of variables. </span><span style="font-family: Helvetica; font-size: 12px;" class="">An environment can be passed either by name from the DB or adhoc by dict in start_workflow. </span><span style="font-family: Helvetica; font-size: 12px;" class="">On workflow execution, a copy of the environment is saved with the execution object. </span><span style="font-family: Helvetica; font-size: 12px;" class="">Action inputs are still declared explicitly in the WF spec. This does not solve the problem where common inputs are specified over and over again. So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task. </span><span style="font-family: Helvetica; font-size: 12px;" class="">In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs. The WF author can specify that the conn_str input is supplied from the $.env.conn_str.</span></div><div class=""><br class=""></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><div class=""><u class="">Example:</u></div></div><div class=""><div class=""><br class=""></div></div><div class=""><div class=""><div class=""># Assume this SqlAction is registered as std.sql in Mistral's Action table.</div></div></div><div class=""><div class=""><div class=""><div class="">class SqlAction(object):</div></div></div></div><div class=""><div class=""><div class=""><div class=""> def __init__(self, conn_str, query):</div></div></div></div><div class=""><div class=""><div class=""><div class=""> ...</div></div></div></div><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div><div class=""><div class=""><div class=""><div class="">...</div></div></div></div><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div><div class=""><div class=""><div class=""><div class="">version: "2.0"</div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;">workflows:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> demo:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> type: direct</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> input:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> - query</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> output:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> - records</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> tasks:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> query:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> action: std.sql conn_str={$.env.conn_str} query={$.query}</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> publish:</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"> records: $</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class="">...</div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px;" class="">my_adhoc_env = {</span></div></div><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px;" class=""> "conn_str": "</span><a href="mysql://admin:secrete@" class="" style="font-family:Helvetica;font-size:12px">mysql://admin:secrete</a>@localhost<span style="font-family: Helvetica; font-size: 12px;" class="">/test</span><span style="font-family: Helvetica; font-size: 12px;" class="">"</span></div></div><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px;" class="">}</span></div></div><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px;" class=""><br class=""></span></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;">...</div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><br class=""></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""># adhoc by dict</div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class="">start_workflow(wf_name, wf_inputs, env=my_adhoc_env)</div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><br class=""></div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class="">OR</div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><br class=""></div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""># lookup by name from DB model</div></div></div></div><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class="">start_workflow(wf_name, wf_inputs, env="my_lab_env")</div></div></div></div></blockquote><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px;" class=""><br class=""></span></div><div class=""><span class="" style="font-family: Helvetica; font-size: 12px;"><b class="">Define Default Action Inputs as Environment Variables</b><br class=""></span><span class="" style="font-family: Helvetica; font-size: 12px;">Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again. </span><span style="font-family: Helvetica; font-size: 12px;" class="">On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment. </span><span style="font-family: Helvetica; font-size: 12px;" class=""> </span><div class="" style="font-family: Helvetica; font-size: 12px;"><span class=""><br class=""></span></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><span class=""><span class=""><u class="">Example:</u></span></span></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class="">Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec. However, the author also has the option to supply that as default action inputs. An example environment structure is below. "__actions" should be reserved and immutable. Users can specific one or more default inputs for the sql action as nested dict under "__actions". Recursive YAQL eval should be supported in the env variables.</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class="">version: "2.0"</div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class="">workflows:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> demo:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> type: direct</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> input:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> - query</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> output:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> - records</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> tasks:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> query:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> action: std.sql query={$.query}</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> publish:</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""> records: $</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><div class="">...</div></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class="">my_adhoc_env = {</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> "sql_server": "localhost",</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> "__actions": {</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> "std.sql": {</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> "conn_str": "<a href="mysql://admin:secrete@" class="">mysql://admin:secrete@</a>{$.env.sql_server}/test"</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> }</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""> }</div></div></div></div></div><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class="">}</div></div></div></div></div></blockquote><div class=""><div class="" style="font-family: Helvetica; font-size: 12px;"><div class=""><div class=""><div class=""><br class=""></div><div class=""><b class="">Default Input Values Supplied Explicitly in WF Spec</b></div><div class="">Please refer to this <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values" class="">blueprint</a> for background. This is a different use case. To support, we just need to set the correct order of precedence in applying values.</div><div class=""><div class="" style="word-wrap:break-word">1. Input explicitly given to the sub flow/task in the WF spec</div><div class="" style="word-wrap:break-word">2. Default input supplied from env</div><div class="" style="word-wrap:break-word">3. Default input supplied at WF spec</div><div class="" style="word-wrap:break-word"><br class=""></div></div><div class=""><b class="">Putting this together...</b></div><div class="">At runtime, the WF context would be similar to the following example. This will be use to recursively eval the inputs for subflow/tasks/actions. </div><div class=""><br class=""></div><div class=""><div class="">ctx = {</div><div class=""> "var1": …,</div><div class=""> "var2": …,</div><div class=""> "my_server_ip": "10.1.23.250",</div><div class=""> "env": {</div><div class=""><div class=""> "sql_server": "localhost",</div><div class=""> "__actions": {<br class=""> "std.sql": {<br class=""> "conn": "<a href="mysql://admin:secrete@{$.env.sql_server}/test" class="">mysql://admin:secrete@{$.env.sql_server}/test</a>"<br class=""> },</div><div class=""> "my.action": {</div><div class=""> "endpoint": "http://{$.my_server_ip}/v1/foo"</div><div class=""> }</div><div class=""> }</div></div><div class=""> }</div><div class="">}</div></div><div class=""><br class=""></div><div class=""><div class="" style="word-wrap:break-word"><b class="">Runtime Context</b></div><div class="" style="word-wrap:break-word">Please refer to this <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html" class="">thread</a> for the background and discussion. The only change here is that on run_action, we will pass the runtime data as kwargs to all action invocation. This means all Action classes should have at least **kwargs added to the __init__ method.<br class=""></div><div class="" style="word-wrap:break-word"><br class=""></div><div class="" style="word-wrap:break-word">runtime = {</div><div class="" style="word-wrap:break-word"> "execution_id": ...,</div><div class="" style="word-wrap:break-word"> "task_id": ...,</div><div class="" style="word-wrap:break-word"> ...</div><div class="" style="word-wrap:break-word">}</div><div class="" style="word-wrap:break-word"><br class=""></div><div class="" style="word-wrap:break-word">...</div><div class="" style="word-wrap:break-word"><br class=""></div><div class="" style="word-wrap:break-word">action = SomeAction(..., runtime=runtime)</div><div class="" style="word-wrap:break-word"><br class=""></div><div class=""><br class=""></div></div><div class=""><b class=""><br class=""></b></div></div></div></div></div></div>
_______________________________________________<br class="">OpenStack-dev mailing list<br class=""><a href="mailto:OpenStack-dev@lists.openstack.org" class="">OpenStack-dev@lists.openstack.org</a><br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br class=""></div></blockquote></div><br class=""></div></body></html>