Open Stack

Thanks Winson,

Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job!

However, all additional comments and questions are more than welcomed!


Renat Akhmerov
@ Mirantis Inc.



> On 24 Dec 2014, at 04:32, W Chan <m4d.coder at gmail.com> wrote:
> 
> After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints.
> * https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment <https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment>
> * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context <https://blueprints.launchpad.net/mistral/+spec/mistral-global-context>
> * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values>
> * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context <https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context>
> 
> Please refer to the following threads for backgrounds.
> * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html>
> * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html>
> * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html>
> 
> 
> Workflow Context Scope
> 1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs
> 2. context are passed by value (copy.deepcopy) to children
> 3. change to context is passed to parent only when it's explicitly published at the end of the child execution
> 4. change to context at the parent (after a publish from a child) is passed to subsequent children
> 
> Environment Variables
> Solves the problem for quickly passing pre-defined inputs to a WF execution.  In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc.  We should implement an API and DB model where users can pre-defined different environments with their own set of variables.  An environment can be passed either by name from the DB or adhoc by dict in start_workflow.  On workflow execution, a copy of the environment is saved with the execution object.  Action inputs are still declared explicitly in the WF spec.  This does not solve the problem where common inputs are specified over and over again.  So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task.  In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs.  The WF author can specify that the conn_str input is supplied from the $.env.conn_str.
> 
> Example:
> 
> # Assume this SqlAction is registered as std.sql in Mistral's Action table.
> class SqlAction(object):
>     def __init__(self, conn_str, query):
>         ...
> 
> ...
> 
> version: "2.0"
> workflows:
>     demo:
>         type: direct
>         input:
>             - query
>         output:
>             - records
>         tasks:
>             query:
>                 action: std.sql conn_str={$.env.conn_str} query={$.query}
>                 publish:
>                     records: $
> 
> ...
> 
> my_adhoc_env = {
>     "conn_str": "mysql://admin:secrete <mysql://admin:secrete@>@localhost/test"
> }
> 
> ...
> 
> # adhoc by dict
> start_workflow(wf_name, wf_inputs, env=my_adhoc_env)
> 
> OR
> 
> # lookup by name from DB model
> start_workflow(wf_name, wf_inputs, env="my_lab_env")
> 
> Define Default Action Inputs as Environment Variables
> Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again.  On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment.    
> 
> Example:
> Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec.  However, the author also has the option to supply that as default action inputs.  An example environment structure is below.  "__actions" should be reserved and immutable.  Users can specific one or more default inputs for the sql action as nested dict under "__actions".  Recursive YAQL eval should be supported in the env variables.
> 
> version: "2.0"
> workflows:
>     demo:
>         type: direct
>         input:
>             - query
>         output:
>             - records
>         tasks:
>             query:
>                 action: std.sql query={$.query}
>                 publish:
>                     records: $
> 
> ...
> 
> my_adhoc_env = {
>     "sql_server": "localhost",
>     "__actions": {
>         "std.sql": {
>             "conn_str": "mysql://admin:secrete@ <mysql://admin:secrete@>{$.env.sql_server}/test"
>          }
>     }
> }
> 
> Default Input Values Supplied Explicitly in WF Spec
> Please refer to this blueprint <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values> for background.  This is a different use case.  To support, we just need to set the correct order of precedence in applying values.
> 1. Input explicitly given to the sub flow/task in the WF spec
> 2. Default input supplied from env
> 3. Default input supplied at WF spec
> 
> Putting this together...
> At runtime, the WF context would be similar to the following example.  This will be use to recursively eval the inputs for subflow/tasks/actions.  
> 
> ctx = {
>    "var1": …,
>    "var2": …,
>    "my_server_ip": "10.1.23.250",
>    "env": {
>         "sql_server": "localhost",
>         "__actions": {
>             "std.sql": {
>                 "conn": "mysql://admin:secrete@{$.env.sql_server}/test <mysql://admin:secrete@{$.env.sql_server}/test>"
>             },
>             "my.action": {
>                 "endpoint": "http://{$.my_server_ip}/v1/foo"
>             }
>         }
>    }
> }
> 
> Runtime Context
> Please refer to this thread <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html> for the background and discussion.  The only change here is that on run_action, we will pass the runtime data as kwargs to all action invocation.  This means all Action classes should have at least **kwargs added to the __init__ method.
> 
> runtime = {
>    "execution_id": ...,
>    "task_id": ...,
>    ...
> }
> 
> ...
> 
> action = SomeAction(..., runtime=runtime)
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141224/8680275e/attachment-0001.html>