<div dir="ltr"><div>Winson, Renat,</div><div><br></div><div>I have a few questions, because some of aspects aren't clear to me.</div><div><br></div><div>1) How does the end user will pass env variables to workflow?Will you add one more optional parameter to execution-create command? </div><div><font face="monospace, monospace">mistral execution-create wf wf_input wf_params wf_env</font></div><div>If yes than what is wf_env will be, json file?</div><div>2) Retuning to first example:</div><div>...</div><div> action: std.sql conn_str={$.env.conn_str} query={$.query}</div><div>...</div><div>$.env - is it a name of environment or it will be a registered syntax to getting access to values from env ?</div><div>3) Can user has a few environments?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 24, 2014 at 8:20 AM, Renat Akhmerov <span dir="ltr"><<a href="mailto:rakhmerov@mirantis.com" target="_blank">rakhmerov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Thanks Winson,<div><br></div><div>Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job!</div><div><br></div><div>However, all additional comments and questions are more than welcomed!</div><div><br></div><div><br><div>
<div>Renat Akhmerov</div><div>@ Mirantis Inc.</div><div><br></div><br>
</div>
<br><div><blockquote type="cite"><div><div class="h5"><div>On 24 Dec 2014, at 04:32, W Chan <<a href="mailto:m4d.coder@gmail.com" target="_blank">m4d.coder@gmail.com</a>> wrote:</div><br></div></div><div><div><div class="h5"><div dir="ltr"><div><div>After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints.</div><div>* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment" target="_blank">https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment</a><br></div><div>* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-global-context" target="_blank">https://blueprints.launchpad.net/mistral/+spec/mistral-global-context</a></div><div>* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values" target="_blank">https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values</a></div><div>* <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context" target="_blank">https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context</a></div></div><div><br></div><div>Please refer to the following threads for backgrounds.</div><div><div>* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html</a></div><div>* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html</a><br></div></div><div>* <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html</a><br></div><div><br></div><div><br></div><div><div><div style="font-family:Helvetica;font-size:12px"><div><b>Workflow Context Scope</b></div><div>1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs<br></div><div>2. context are passed by value (copy.deepcopy) to children</div><div>3. change to context is passed to parent only when it's explicitly published at the end of the child execution</div><div>4. change to context at the parent (after a publish from a child) is passed to subsequent children</div></div><div style="font-family:Helvetica;font-size:12px"><br></div><span style="font-family:Helvetica;font-size:12px"><b>Environment Variables</b></span></div><div><font face="Helvetica">Solves the problem for quickly passing pre-defined inputs to a WF execution. In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc. </font><span style="font-family:Helvetica;font-size:12px">We should implement an API and DB model where users can pre-defined different environments with their own set of variables. </span><span style="font-family:Helvetica;font-size:12px">An environment can be passed either by name from the DB or adhoc by dict in start_workflow. </span><span style="font-family:Helvetica;font-size:12px">On workflow execution, a copy of the environment is saved with the execution object. </span><span style="font-family:Helvetica;font-size:12px">Action inputs are still declared explicitly in the WF spec. This does not solve the problem where common inputs are specified over and over again. So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task. </span><span style="font-family:Helvetica;font-size:12px">In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs. The WF author can specify that the conn_str input is supplied from the $.env.conn_str.</span></div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><u>Example:</u></div></div><div><div><br></div></div><div><div><div># Assume this SqlAction is registered as std.sql in Mistral's Action table.</div></div></div><div><div><div><div>class SqlAction(object):</div></div></div></div><div><div><div><div> def __init__(self, conn_str, query):</div></div></div></div><div><div><div><div> ...</div></div></div></div><div><div><div><div><br></div></div></div></div><div><div><div><div>...</div></div></div></div><div><div><div><div><br></div></div></div></div><div><div><div><div>version: "2.0"</div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px">workflows:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> demo:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> type: direct</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> input:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> - query</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> output:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> - records</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> tasks:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> query:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> action: std.sql conn_str={$.env.conn_str} query={$.query}</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> publish:</div></div></div></div></div><div><div><div><div><div style="font-family:Helvetica;font-size:12px"> records: $</div></div></div></div></div><div><div><div><div><div><br></div></div></div></div></div><div><div><div><div><div>...</div></div></div></div></div><div><div><div><div><div><br></div></div></div></div></div><div><div><span style="font-family:Helvetica;font-size:12px">my_adhoc_env = {</span></div></div><div><div><span style="font-family:Helvetica;font-size:12px"> "conn_str": "</span><a style="font-family:Helvetica;font-size:12px">mysql://admin:secrete</a>@localhost<span style="font-family:Helvetica;font-size:12px">/test</span><span style="font-family:Helvetica;font-size:12px">"</span></div></div><div><div><span style="font-family:Helvetica;font-size:12px">}</span></div></div><div><div><span style="font-family:Helvetica;font-size:12px"><br></span></div></div><div><div><div style="font-family:Helvetica;font-size:12px">...</div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><br></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div># adhoc by dict</div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div>start_workflow(wf_name, wf_inputs, env=my_adhoc_env)</div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div><br></div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div>OR</div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div><br></div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div># lookup by name from DB model</div></div></div></div><div><div><div style="font-family:Helvetica;font-size:12px"><div>start_workflow(wf_name, wf_inputs, env="my_lab_env")</div></div></div></div></blockquote><div><div><span style="font-family:Helvetica;font-size:12px"><br></span></div><div><span style="font-family:Helvetica;font-size:12px"><b>Define Default Action Inputs as Environment Variables</b><br></span><span style="font-family:Helvetica;font-size:12px">Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again. </span><span style="font-family:Helvetica;font-size:12px">On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment. </span><span style="font-family:Helvetica;font-size:12px"> </span><div style="font-family:Helvetica;font-size:12px"><span><br></span></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div style="font-family:Helvetica;font-size:12px"><span><span><u>Example:</u></span></span></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div>Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec. However, the author also has the option to supply that as default action inputs. An example environment structure is below. "__actions" should be reserved and immutable. Users can specific one or more default inputs for the sql action as nested dict under "__actions". Recursive YAQL eval should be supported in the env variables.</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><br></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div>version: "2.0"</div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div>workflows:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> demo:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> type: direct</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> input:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> - query</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> output:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> - records</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> tasks:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> query:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> action: std.sql query={$.query}</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> publish:</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div> records: $</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div><br></div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><div>...</div></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><div><br></div></div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div>my_adhoc_env = {</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> "sql_server": "localhost",</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> "__actions": {</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> "std.sql": {</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> "conn_str": "<a>mysql://admin:secrete@</a>{$.env.sql_server}/test"</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> }</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div> }</div></div></div></div></div><div><div style="font-family:Helvetica;font-size:12px"><div><div><div>}</div></div></div></div></div></blockquote><div><div style="font-family:Helvetica;font-size:12px"><div><div><div><br></div><div><b>Default Input Values Supplied Explicitly in WF Spec</b></div><div>Please refer to this <a href="https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values" target="_blank">blueprint</a> for background. This is a different use case. To support, we just need to set the correct order of precedence in applying values.</div><div><div style="word-wrap:break-word">1. Input explicitly given to the sub flow/task in the WF spec</div><div style="word-wrap:break-word">2. Default input supplied from env</div><div style="word-wrap:break-word">3. Default input supplied at WF spec</div><div style="word-wrap:break-word"><br></div></div><div><b>Putting this together...</b></div><div>At runtime, the WF context would be similar to the following example. This will be use to recursively eval the inputs for subflow/tasks/actions. </div><div><br></div><div><div>ctx = {</div><div> "var1": …,</div><div> "var2": …,</div><div> "my_server_ip": "10.1.23.250",</div><div> "env": {</div><div><div> "sql_server": "localhost",</div><div> "__actions": {<br> "std.sql": {<br> "conn": "<a>mysql://admin:secrete@{$.env.sql_server}/test</a>"<br> },</div><div> "my.action": {</div><div> "endpoint": "http://{$.my_server_ip}/v1/foo"</div><div> }</div><div> }</div></div><div> }</div><div>}</div></div><div><br></div><div><div style="word-wrap:break-word"><b>Runtime Context</b></div><div style="word-wrap:break-word">Please refer to this <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html" target="_blank">thread</a> for the background and discussion. The only change here is that on run_action, we will pass the runtime data as kwargs to all action invocation. This means all Action classes should have at least **kwargs added to the __init__ method.<br></div><div style="word-wrap:break-word"><br></div><div style="word-wrap:break-word">runtime = {</div><div style="word-wrap:break-word"> "execution_id": ...,</div><div style="word-wrap:break-word"> "task_id": ...,</div><div style="word-wrap:break-word"> ...</div><div style="word-wrap:break-word">}</div><div style="word-wrap:break-word"><br></div><div style="word-wrap:break-word">...</div><div style="word-wrap:break-word"><br></div><div style="word-wrap:break-word">action = SomeAction(..., runtime=runtime)</div><div style="word-wrap:break-word"><br></div><div><br></div></div><div><b><br></b></div></div></div></div></div></div></div></div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br></div></blockquote></div><br></div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>