Open Stack

Winson, Renat,

I have a few questions, because some of aspects aren't clear to me.

1) How does the end user will pass env variables to workflow?Will you add
one more optional parameter to execution-create command?
mistral execution-create wf wf_input wf_params wf_env
If yes than what is wf_env will be, json file?
2) Retuning to first example:
...
 action: std.sql conn_str={$.env.conn_str} query={$.query}
...
$.env - is it a name of environment or it will be a registered syntax to
getting access to values from env ?
3) Can user has a few environments?

On Wed, Dec 24, 2014 at 8:20 AM, Renat Akhmerov <rakhmerov at mirantis.com>
wrote:

> Thanks Winson,
>
> Since we discussed all this already I just want to confirm that I fully
> support this model, it will significantly help us make much more concise,
> readable and maintainable workflows. I spent a lot of time thinking about
> it and don’t see any problems with it. Nice job!
>
> However, all additional comments and questions are more than welcomed!
>
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
>
> On 24 Dec 2014, at 04:32, W Chan <m4d.coder at gmail.com> wrote:
>
> After some online discussions with Renat, the following is a revision of
> the proposal to address the following related blueprints.
> *
> https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment
> * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context
> *
> https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values
> * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context
>
> Please refer to the following threads for backgrounds.
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html
>
>
> *Workflow Context Scope*
> 1. context to workflow is passed to all its subflows and subtasks/actions
> (aka children) only explicitly via inputs
> 2. context are passed by value (copy.deepcopy) to children
> 3. change to context is passed to parent only when it's explicitly
> published at the end of the child execution
> 4. change to context at the parent (after a publish from a child) is
> passed to subsequent children
>
> *Environment Variables*
> Solves the problem for quickly passing pre-defined inputs to a WF
> execution.  In the WF spec, environment variables are referenced as
> $.env.var1, $.env.var2, etc.  We should implement an API and DB model
> where users can pre-defined different environments with their own set of
> variables.  An environment can be passed either by name from the DB or
> adhoc by dict in start_workflow.  On workflow execution, a copy of the
> environment is saved with the execution object.  Action inputs are still
> declared explicitly in the WF spec.  This does not solve the problem where
> common inputs are specified over and over again.  So if there are multiple
> SQL tasks in the WF, the WF author still needs to supply the conn_str
> explicitly for each task.  In the example below, let's say we have a SQL
> Query Action that takes a connection string and a query statement as
> inputs.  The WF author can specify that the conn_str input is supplied from
> the $.env.conn_str.
>
> *Example:*
>
> # Assume this SqlAction is registered as std.sql in Mistral's Action table.
> class SqlAction(object):
>     def __init__(self, conn_str, query):
>         ...
>
> ...
>
> version: "2.0"
> workflows:
>     demo:
>         type: direct
>         input:
>             - query
>         output:
>             - records
>         tasks:
>             query:
>                 action: std.sql conn_str={$.env.conn_str} query={$.query}
>                 publish:
>                     records: $
>
> ...
>
> my_adhoc_env = {
>     "conn_str": "mysql://admin:secrete@localhost/test"
> }
>
> ...
>
> # adhoc by dict
> start_workflow(wf_name, wf_inputs, env=my_adhoc_env)
>
> OR
>
> # lookup by name from DB model
> start_workflow(wf_name, wf_inputs, env="my_lab_env")
>
>
> *Define Default Action Inputs as Environment Variables*
> Solves the problem where we're specifying the same inputs to subflows and
> subtasks/actions over and over again.  On command execution, if action
> inputs are not explicitly supplied, then defaults will be lookup from the
> environment.
>
> *Example:*
> Using the same example from above, the WF author can still supply both
> conn_str and query inputs in the WF spec.  However, the author also has the
> option to supply that as default action inputs.  An example environment
> structure is below.  "__actions" should be reserved and immutable.  Users
> can specific one or more default inputs for the sql action as nested dict
> under "__actions".  Recursive YAQL eval should be supported in the env
> variables.
>
> version: "2.0"
> workflows:
>     demo:
>         type: direct
>         input:
>             - query
>         output:
>             - records
>         tasks:
>             query:
>                 action: std.sql query={$.query}
>                 publish:
>                     records: $
>
> ...
>
> my_adhoc_env = {
>     "sql_server": "localhost",
>     "__actions": {
>         "std.sql": {
>             "conn_str": "mysql://admin:secrete@{$.env.sql_server}/test"
>          }
>     }
> }
>
>
> *Default Input Values Supplied Explicitly in WF Spec*
> Please refer to this blueprint
> <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values>
> for background.  This is a different use case.  To support, we just need to
> set the correct order of precedence in applying values.
> 1. Input explicitly given to the sub flow/task in the WF spec
> 2. Default input supplied from env
> 3. Default input supplied at WF spec
>
> *Putting this together...*
> At runtime, the WF context would be similar to the following example.
> This will be use to recursively eval the inputs for subflow/tasks/actions.
>
> ctx = {
>    "var1": …,
>    "var2": …,
>    "my_server_ip": "10.1.23.250",
>    "env": {
>         "sql_server": "localhost",
>         "__actions": {
>             "std.sql": {
>                 "conn": "mysql://admin:secrete@{$.env.sql_server}/test"
>             },
>             "my.action": {
>                 "endpoint": "http://{$.my_server_ip}/v1/foo"
>             }
>         }
>    }
> }
>
> *Runtime Context*
> Please refer to this thread
> <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html> for
> the background and discussion.  The only change here is that on run_action,
> we will pass the runtime data as kwargs to all action invocation.  This
> means all Action classes should have at least **kwargs added to the
> __init__ method.
>
> runtime = {
>    "execution_id": ...,
>    "task_id": ...,
>    ...
> }
>
> ...
>
> action = SomeAction(..., runtime=runtime)
>
>
>
>  _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141224/6d7e336a/attachment.html>