We are delighted to announce the release of: mistral 9.0.0: Mistral Project This release is part of the train release series. The source is available from: https://opendev.org/openstack/mistral Download the package from: https://pypi.org/project/mistral Please report issues through: https://bugs.launchpad.net/mistral/+bugs For more details, please see below. Changes in mistral 8.0.0.0rc1..9.0.0 ------------------------------------ 016509eb Update TOX/UPPER_CONSTRAINTS_FILE for stable/train 33d4a8bb Fix "root_execution" lazy loading issue and refactor execution.py aae80641 Update .gitreview for stable/train a655ff98 Add the explicit dependency from kombu ac41f94d Add an ability to disable workflow text validation 6fb83388 Fix scheduled jobs migration b4a46fd1 Remove volumes.promote and volumes.reenable action from cinder 0de24794 Pass a real session to ironicclient in _get_fake_client b9b21c82 Exclude ironicclient 3.0.0 from requirements 64c4038b Optimize creation of language specs efc4cf07 Use v2 designate client instead of v1 86ae8a7e Change 403 exception message to something more informative and accurate 8283670d Remove unneeded Zuul branch matcher 7e7f1cb9 moved generic util functions from mistral to mistral-lib 51b7dd0c Fix error validate token when run cron trigger 9585a638 Fix don't work with senlin actions 6418132a Fix missing in workflow documents 4a1be8c4 Add db api tests for scheduled jobs 24f50dab Need to run only doc related jobs on doc change f13b69ca Fix workflow documents 21f95a58 Add a migration to create the scheduled_jobs table fdfb65a8 Fix workflow language tests after updating jsonschema to 3.0.2 7158d90b Fix misspell word 6a584299 Blacklist eventlet 0.21.0,0.23.0,0.25.0 c99b87a8 Check if workflow execution is empty in integrity checker 9804e74d Add "published_global" field to the task execution REST resource 5eb2a216 Improve workflow notifications and webhook data f92a5c8f Fix 'with-items' expression evaluation 6102c525 Mistral don't work with cron trigger 6dfab10e Support OpenStack services dynamic versions 60536f69 Bump lower constraint of python-zunclient 8d539921 remove unused gate code * post_test_hook. used in legacy gate * run_functional_test probably not used and also doean't work since mistral_tempest was moved 6c0bd2a2 Add "retry_count" field into workflow execution report 0f6bc189 Improve new scheduler 4e926a1f Fail-on policy 059248f8 Create docs for the workflow namespaces feature 9b576aec Use raw strings in all pattern matching strings d8a127ab Upgrade hacking babcd0d6 Improve error message when sync actions timeout 12c84dc9 Allow to filter event notifications by their type 7ae3ec0c Provide better docs for the "target" task attribute 1a2ff637 A1 B should be tasks rather than workflow levels 2c3c3b1b Update api-ref location 77118d3b Rename the test class so it will be PyCharm friendly 90a9a48f When installing devstack enable the bash completion 4a9d55a1 Retry a DB transaction on "Too many connections" error 69148233 Bulk delete of stored delayed calls f43bf7ba Allow to delete multiple objects with advanced filters 6098804a remove mistral-tempest-plugin from setup.cfg 4539a0a0 Fix execution deletion in case of insecure context aed90bab New rerun events c078429d Add workflow_execution_id to task notification object b6ebcb3b Use SessionClient for Ironic actions 843717a1 Add Python 3 Train unit tests 93dce86f Fix invalid assert states 26f6e192 Add bindep.txt file for binary dependencies used in unit tests dfc3277f Initialize the lazily loaded execution "input" field in API 43d23c0e Create needed infrastructure to switch scheduler implementations f5f9a1df Fix workflow execution cascade delete error b97b2485 Use eventlet-aware threading events 0c3b6306 Make more JSON fields in execution objects lazy-loaded b0fb101c Optimize finding upstream task executions c215c052 Direct workflow code cleanup and refactoring 5c5e948d Use openstack-python3-train-jobs for python3 test runtime aed4b18c Fix sporadically failing test test_with_items_action_context b6947705 Store next task names in DB eb593285 Limit max search depth a0cada98 Change mistral opendev.org to releases.openstack.org 2a1063e7 Add python 3.7 classifier to setup.cfg 9a2f3b57 Simple optimization of creating/updating workflows bea98f6f Reduce number of fields in notification data 475b82c5 Delete delayed calls for deleted entities 1e83a076 Exclude broken ironicclient versions 2.5.2 and 2.7.1 4e391dc2 Handle action inputs properly to prevent tasks stuck in RUNNING state 5f86cf1f Allow to use 'task()' yaql function in task policies 15355ea5 Use get_task_executions_count for any_cancels method e5031e26 Rework updating action executions heartbeats 6dc981d8 Bump openstackdocstheme to 1.30.0 0731cd1d Blacklist sphinx 2.1.0 (autodoc bug) f09c8ebe Skip context evaluation for non-conditional transitions c1e4fd8d Remove _get_next_clauses 9289f7d2 Add release notes to changes to wflanguage execution object 58b714eb Prepare cache for _is_upstream_task_execution cd19e486 Remove _find_task_execution_by_name 92528a82 Move action caching to db layer a75b07ec Send task and workflow notifications out of the transaction c5268852 Rework finding indirectly affected created joins 2712b8c1 Fix adhoc action lookup 424a0a55 Blacklist python-cinderclient 4.0.0 831162ac Constraint networkx to <2.3 for Python 2 84b8e92a Get rid of lookup utils a477fe49 Fix how "has_next_tasks" is calculated for task executions 8549aeaf Optimize searching of upstream task executions ff00c9c7 Rework joining mechanism dbc4c911 Remove deprecated nova commands 83c541ac Reduce the number of "on-xxx" evaluations eb592162 Removes insecure parameter from barbican client 7e0f4eee Add back the secrets_store action into mapping.json f172e5da Adding root_execution_id to the jinja executor function a83b4a67 Fix the import for filter_utils b7107c84 Adds secrets_retrieve to the list of available actions in barbican 1df722d9 Pass auth context to the event publishers 09cd21d5 Docs improvements: task timeout, global context, Docker and jinja 13d8f532 Docs: Change description from on-success to on-error 88e5af41 Reformat retry logic for tasks with join 243c09d5 Add option to start subworkflow via RPC. 46fd1b8a Replace git.openstack.org URLs with opendev.org URLs f9f99475 Add delay option to std.echo to emulate external lags. a461e079 OpenDev Migration Patch 279fcc5b Add release notes for reply-to feature 9a60cd62 Fix an action execution controller test 97b493ad Don't use default mutable parameter 59a96b73 inspect.getargspec is deprecated on py3 2cbf5431 Add reply-to to std.email c5808e54 Dropping the py35 testing 401eeddb Fix E305 codestyle errors 1af2683d Drop py35 jobs 9c65b7d2 Fix doc b0829f94 Fix an expression context for all_errors_handled() 20c2b78d Fix an obsolete name of a profiler trace 1fce28c1 Add a script to generate a report about profile traces 5d2be345 Add release note for I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36 765225b1 Replace openstack.org git:// URLs with https:// 9238909e Stop sending workflow output to on_action_complete 5e0133c8 Add "convert_input_data" config property for YAQL expressions 97f4fc27 Setting the lower version of sphinxcontrib-pecanwsme to 0.10.0 d332d7fe Add release notes for engine optimizations d4a4f7a5 Improve profiler logging 3e42cffd Update master for stable/stein 218ab5a7 Add Python 3.7 to tox 32c96b1b Add "root_execution" mapped property to WorkflowExecution model a73e3def Optimize action scheduling ca1acb65 Add http_proxy_to_wsgi middleware 9bbc1d41 Adds private_key parameter in the standard ssh actions Diffstat (except docs and test files) ------------------------------------- .gitreview | 3 +- .zuul.yaml | 34 +- CONTRIBUTING.rst | 12 +- bindep.txt | 8 + devstack/plugin.sh | 9 +- devstack/settings | 14 +- lower-constraints.txt | 24 +- mistral/actions/openstack/action_generator/base.py | 2 +- mistral/actions/openstack/actions.py | 157 +++++--- mistral/actions/openstack/base.py | 2 +- mistral/actions/openstack/mapping.json | 11 +- mistral/actions/std_actions.py | 45 ++- mistral/api/app.py | 4 + mistral/api/controllers/resource.py | 2 +- mistral/api/controllers/root.py | 2 +- mistral/api/controllers/v2/environment.py | 2 +- mistral/api/controllers/v2/execution.py | 45 ++- mistral/api/controllers/v2/execution_report.py | 5 + mistral/api/controllers/v2/member.py | 2 + mistral/api/controllers/v2/resources.py | 9 +- mistral/api/controllers/v2/root.py | 3 +- mistral/api/controllers/v2/task.py | 65 ++- mistral/api/controllers/v2/workbook.py | 17 +- mistral/api/controllers/v2/workflow.py | 27 +- mistral/api/hooks/content_type.py | 2 +- mistral/cmd/launch.py | 2 +- mistral/config.py | 58 ++- mistral/context.py | 4 +- mistral/db/sqlalchemy/base.py | 2 +- .../027_add_last_heartbeat_to_action_execution.py | 2 +- ...xt_tasks_and_error_handled_to_task_execution.py | 40 ++ .../033_add_next_tasks_to_task_execution.py | 38 ++ .../versions/034_add_scheduled_jobs_table.py | 68 ++++ mistral/db/sqlalchemy/migration/cli.py | 1 + mistral/db/sqlalchemy/model_base.py | 2 +- mistral/db/utils.py | 19 +- mistral/db/v2/api.py | 39 +- mistral/db/v2/sqlalchemy/api.py | 92 ++++- mistral/db/v2/sqlalchemy/models.py | 42 +- mistral/engine/action_handler.py | 23 +- mistral/engine/actions.py | 103 +++-- mistral/engine/base.py | 6 +- mistral/engine/default_engine.py | 28 +- mistral/engine/engine_server.py | 14 +- mistral/engine/policies.py | 149 ++++--- mistral/engine/post_tx_queue.py | 2 +- mistral/engine/task_handler.py | 90 +++-- mistral/engine/tasks.py | 98 +++-- mistral/engine/utils.py | 2 +- mistral/engine/workflow_handler.py | 27 +- mistral/engine/workflows.py | 89 +++-- mistral/exceptions.py | 2 +- mistral/executors/executor_server.py | 2 +- mistral/expressions/yaql_expression.py | 2 +- mistral/lang/base.py | 99 +++-- mistral/lang/parser.py | 4 +- mistral/lang/types.py | 6 +- mistral/lang/v2/actions.py | 2 +- mistral/lang/v2/on_clause.py | 8 +- mistral/lang/v2/policies.py | 8 +- mistral/lang/v2/publish.py | 2 +- mistral/lang/v2/retry_policy.py | 2 +- mistral/lang/v2/task_defaults.py | 6 +- mistral/lang/v2/tasks.py | 8 +- mistral/lang/v2/workbook.py | 2 +- mistral/lang/v2/workflows.py | 9 +- mistral/messaging.py | 2 +- mistral/notifiers/base.py | 2 +- mistral/notifiers/default_notifier.py | 7 +- mistral/notifiers/notification_events.py | 12 +- mistral/notifiers/notification_server.py | 2 +- mistral/notifiers/publishers/noop.py | 2 +- mistral/notifiers/publishers/webhook.py | 2 +- mistral/rpc/base.py | 9 + mistral/rpc/clients.py | 9 +- mistral/rpc/kombu/kombu_client.py | 2 +- mistral/rpc/kombu/kombu_listener.py | 3 +- mistral/rpc/kombu/kombu_server.py | 5 +- mistral/scheduler/base.py | 66 ++- mistral/scheduler/default_scheduler.py | 83 +++- mistral/service/coordination.py | 2 +- mistral/services/__init__.py | 29 ++ mistral/services/action_execution_checker.py | 2 +- mistral/services/action_manager.py | 2 +- .../services/{scheduler.py => legacy_scheduler.py} | 90 ++--- mistral/services/workbooks.py | 21 +- mistral/services/workflows.py | 56 ++- ...test_lookup_utils.py => test_action_caching.py} | 76 +--- .../unit/engine/test_direct_workflow_rerun.py | 8 +- .../unit/engine/test_reverse_workflow_rerun.py | 17 +- .../unit/engine/test_subworkflows_pause_resume.py | 246 +++++------- .../unit/expressions/test_jinja_expression.py | 10 +- ...test_scheduler.py => test_default_scheduler.py} | 42 +- ...{test_scheduler.py => test_legacy_scheduler.py} | 156 +++++--- mistral/utils/__init__.py | 441 --------------------- mistral/utils/expression_utils.py | 23 +- mistral/utils/inspect_utils.py | 17 +- mistral/utils/openstack/keystone.py | 32 +- mistral/utils/profiler.py | 26 ++ mistral/utils/rest_utils.py | 14 +- mistral/utils/ssh_utils.py | 30 +- mistral/workflow/base.py | 30 +- mistral/workflow/commands.py | 53 ++- mistral/workflow/data_flow.py | 30 +- mistral/workflow/direct_workflow.py | 366 ++++++++--------- mistral/workflow/lookup_utils.py | 190 --------- mistral/workflow/reverse_workflow.py | 31 +- playbooks/legacy/mistral-ha/run.yaml | 2 +- ...oot_execution_id_to_jinja-90b67c69a50370b5.yaml | 5 + .../add_skip_validation-9e8b906c45bdb89f.yaml | 18 + ...nput_data_config_property-09822dee1f46eb8e.yaml | 11 + ...t_tasks_field_calculation-5717f93d7adcd9b0.yaml | 7 + .../notes/fix_task_function-04b83ada20a71f12.yaml | 6 + ...y-to-wsgi-oslo-middleware-f66f1b9533ea1e8a.yaml | 11 + ..._adhoc_actions_scheduling-e324f66f962ae409.yaml | 8 + .../notes/std-ssh-add-pkey-2c665a81ff9fbdfd.yaml | 6 + .../notes/std.email-reply-to-c283770c798db7d0.yaml | 3 + ...entity_for_root_execution-1af6af12ee437282.yaml | 11 + releasenotes/source/index.rst | 1 + releasenotes/source/stein.rst | 6 + requirements.txt | 23 +- setup.cfg | 7 +- test-requirements.txt | 7 +- tools/config/config-generator.mistral.conf | 1 + tools/docker/DOCKER_README.rst | 176 +------- tools/docker/Dockerfile | 2 +- tools/rank_profiled_methods.py | 99 +++++ tox.ini | 8 +- 197 files changed, 4658 insertions(+), 2882 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index f21d2750..22ff0816 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11 +11 @@ dogpile.cache>=0.6.2 # BSD -eventlet!=0.20.1,>=0.20.0 # MIT +eventlet!=0.20.1,!=0.21.0,!=0.23.0,!=0.25.0,>=0.20.0 # MIT @@ -14 +14 @@ Jinja2>=2.10 # BSD License (3 clause) -jsonschema<3.0.0,>=2.6.0 # MIT +#jsonschema>=2.6.0 # MIT @@ -16,2 +16,4 @@ keystonemiddleware>=4.18.0 # Apache-2.0 -mistral-lib>=0.4.0 # Apache-2.0 -networkx>=1.10 # BSD +kombu!=4.0.2,>=4.6.1 # BSD +mistral-lib>=1.2.0 # Apache-2.0 +networkx<2.3,>=1.10;python_version<'3.0' # BSD +networkx>=2.3;python_version>='3.4' # BSD @@ -26 +28 @@ oslo.policy>=1.30.0 # Apache-2.0 -oslo.utils>=3.33.0 # Apache-2.0 +oslo.utils>=3.37.0 # Apache-2.0 @@ -35 +37,2 @@ python-barbicanclient>=4.5.2 # Apache-2.0 -python-cinderclient>=3.3.0 # Apache-2.0 +python-cinderclient!=4.0.0,>=3.3.0 # Apache-2.0 +python-zaqarclient>=1.0.0 # Apache-2.0 @@ -51 +54 @@ python-troveclient>=2.2.0 # Apache-2.0 -python-ironicclient>=2.3.0 # Apache-2.0 +python-ironicclient!=2.7.1,!=3.0.0,>=2.7.0 # Apache-2.0 @@ -54,2 +57 @@ python-vitrageclient>=2.0.0 # Apache-2.0 -python-zaqarclient>=1.0.0 # Apache-2.0 -python-zunclient>=1.0.0 # Apache-2.0 +python-zunclient>=3.4.0 # Apache-2.0 @@ -60 +62 @@ requests>=2.14.2 # Apache-2.0 -tenacity>=4.4.0 # Apache-2.0 +tenacity>=5.0.1 # Apache-2.0 @@ -67,0 +70 @@ zake>=0.1.6 # Apache-2.0 + diff --git a/test-requirements.txt b/test-requirements.txt index c2ce4d6c..bb2db372 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -4 +4 @@ -hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +hacking>=1.1.0,<1.2.0 # Apache-2.0 @@ -11 +11 @@ keystonemiddleware>=4.18.0 # Apache-2.0 -mistral-lib>=0.4.0 # Apache-2.0 +mistral-lib>=1.2.0 # Apache-2.0 @@ -13 +13,2 @@ mock>=2.0.0 # BSD -networkx>=1.10 # BSD +networkx<2.3,>=1.10;python_version<'3.0' # BSD +networkx>=2.3;python_version>='3.4' # BSD