We high-spiritedly announce the release of: mistral 7.0.0: Mistral Project This release is part of the rocky release series. Download the package from: https://pypi.org/project/mistral For more details, please see below. Changes in mistral 6.0.0..7.0.0 ------------------------------- 6ce51e3 Update workbook namespace to '' on migration 2c644bd expose the user info url as a configuration e469400 Fix docker image not building 214828a Keycloak and Docker df46ec7 Clarify what an exception from an action means 3c430ef Improve std.email action 3605022 remove invalid todo comment 66f6c90 Fix flaky cron trigger test 9e2074d Add namespace parameter to Workbook API doc 9f03835 Add a debug log for the webhook publisher 4bf03d8 Remove extra a specification validation 834747b Add namespace support for workbooks d2f03bc Enable mutable config in mistral a2756a3 Support Manila actions in Mistral 36f28de Remove hardcoded usage of v2 authentication in Barbican actions 0867bec Use json.loads instead of eval() on the config e1e6c7a Update the Custom Action documentation to use mistral-lib 38bfd91 Add documentation on event notifier fe6f0c5 Migrate mistral to using the serialization code in mistral-lib dcb4355 Use register_session_conf_options API 4bb9d5d Install mistral-lib as a sibling for tox jobs c08e44f Allow engine commands as task name d3305c7 Release note for adding "oslo_rpc_executor" config option a73fa35 Fix testenv cover in tox.ini 03fbd83 Fix rev-id parameter in mistral-db-manage 549ec1f Return the result of the MistralHTTPAction 30e3062 modify grammar mistake 259d8a8 add docs for states. add docs explaining what each state mean. 0448383 Add the config option for Oslo Messaging executor type f11ddeb Pin get-pip.py to 3.2 69d34b2 Follow the new PTI for document build 522f388 Add a policy to control the right to publish resources 5cfaa9c detect https and act accordingly 2bfc9d1 Add CloudFlow info to Mistral documentation 49ccf8e Add missing Tacker actions to Mistral 60fba72 fix tox python3 overrides d74f7cb Fix typos 981b210 Add better output to the user deleting executions 59e4398 switch mysql functional tests to PyMySQL 8d14c3e Add missing query paramater `scope` to the workbook api dc8b50e Remove time.sleep from tests 821808a Generate default execution id 211e396 Fix mistral CI. 47a293d Update install guide about running mistral 78b542c Refresh a number of retry a task when task was rerun 160948e Use on-clause and retry_policy get_spec for validation 56cf004 Add info about source execution to a workflow execution description 7265217 Amend the spelling error of a word 07c26e5 Make cron-triggers not play catchup ce18c0b A mechanism to close stuck running action executions 968e15e add the release note a8abfae Update default Docker configuration and documentation 96a1bdd Add the restructuredtext check to the flake8 job a7314d2 Fix the call to start_workflow from events triggers d9e4a86 Register the collect_timing keystone_authtoken option 360fd8b Release note for adding YAQL engine options fe0d441 Add YAQL engine options 6a67257 Create Base class for Mistral Exceptions and Errors 9883814 Switch to using stestr ec66e1d Add log message to log the stack trace in case failed to send an HTTP request a1e2ac8 Add test for _try_import 0f65f60 Fix `def _admin` keystone client factory with trust scope 8a8c4bb Replace Chinese quotes with English quotes 3ee4510 Replace port 35357 with 5000 for "identity_uri" d3c6bf7 Release note for using "passive_deletes=True" eadce20 Add "fields" argument to DB API methods 4fe4198 Support Qinling actions in Mistral fac2c4a Added test to update the database schema to the latest revision 696e827 Release note for using "passive_deletes=True" 6359053 Use partial loading of workflow execution in the controller afb0f58 Use "passive_deletes=True" in ORM relationships 02bb7ba Fix workflows query with fields=input c7e33f1 Add namespace in exception info e58f80d Purge README.rst in favor of the generated documentation 2e710e7 Fixed workflow output in case of execution_field_size_limit_kb cb1cabb Release note for workflow environment optimizations f2a9bd4 Do not copy workflow environment into subworkflows b77769c Get rid of a extra copy of workflow environment 11896ba Support actions for zun a70df9e Add release note for jinja expression bug af84fa9 Only allow for deleting completed executions 8c91b17 Remove unused params. 56adb2e Make sure there are no duplicates in the spec cache w/o restarts 92b6016 Fix error about SenlinAction f25fb43 Added Vitrage actions into Mistral Implements: blueprint mistral-vitrage-actions 5f89e2e Added JavaScript evaluator which doesn't require a compilation a29c7d9 Added 'safe-rerun' policy to task-defaults section 94c0a19 Minor `update_on_match` improvement e1f1056 Fixed Jinja error handling d2be248 Update mistral upgrade guide 557c1f5 Stop using slave_scripts/install-distro-packages.sh 15bbcbb Fix docker image build job 5073274 Release note for batched evaluation of final workflow context e7da6af Trivial: Update pypi url to new url 62d1003 Optimize final workflow context evaluation with a batch request b8db6c6 add lower-constraints job e42c515 Release note for not persisting '__task_execution' in DB 6b7b58e Add '__task_execution' structure to task execution context on the fly 15e95d9 Correct the string formatting in a info log message 4936d77 Optimizing big 'on-XXX' clauses e43cf45 Fix DB connection url in config guide 7d8e7db Adding a test for std.test_dict action 96f1f22 Fix call to model_query in sqlalchemy.api._get_collection a6a5310 do not list setuptools as explicit dependency f14f507 uncap eventlet 1ece440 Adding WWW-Authenticate info. be2bfe7 Fix tag searching 1a65bc4 Add Swift Service OpenStack Actions 0d74edd explicitly set the notifier type in unit tests 6de2c15 Fix malformed state_info when json is set to it d456ba2 Update cut_list() to return the specified number of characters e7da5b6 Add a release note for new indexes 82d226a Replace the unsupported format character with the format method e01dade Added thread pool for eventlet executor mode 7657458 Fix join examples and text 95f412a Fix WF execution getting stuck on null description 0090fe2 Update cut_dict() to return no more than specified by length 8b30743 Add a release note for the 'pause' command fix f5c0be6 Fix std.ssh "password" parameter 7bebd3b Updated from global requirements f0b5196 Improve Mistral API DB error handling 9726189 Fix 'pause' engine command e3cb610 Updated from global requirements b8d94fe Change unreasonable title 9b47758 Updated from global requirements 328bb2c Add new indexes to optimize `task().result` expression function e3918a1 Fixed the Mistral Docker build job a59af3f Ensure workflow check is retried on DB error 0ec8ac3 Improve the Docker integration 34c3c45 Fixed a vhost in transport_url processing by kombu driver 6499733 Update Duplicate entry exceptions to provide more information 4e77217 Update test-requirements af0f8a1 Remove unused nova actions 23abbc1 Fix initializers of standard actions 8cb4ff0 Updated from global requirements 6382cf6 Fixed the broken MySQL job f2d8b14 Clarify usage of break-on and continue-on e011feb Remove a redundant initializer of InvalidUnicodeAction in the tests 10990e5 Add a unit test to check initializers of MistralException hierarcy c9139ef Updated from global requirements 6fb10f5 Fix server info string 55990f3 Updated from global requirements 3e04223 Make sure not to swallow original exceptions on handling remote errors 422c89a Add a release note for execution events noitifications dd44e0e Retry DB operation on deadlock 4ae3b1e Updated from global requirements 4a077b6 Fixed the length of a task name 0369fdd Prevent a action completion multiple times 6c7d641 Remove unnecessary locks 629bdcc Rename task_id to task_execution_id 38388a0 Restore rally job ad7d224 Updated from global requirements bd33f0e Updated from global requirements 57ca93f Fix docker image publish job 0232fdb Fix errors of parameter note acbbeac Fix doc format 9ba81eb Replaced the deprecated options in the configuration guide d2ffecd Add the 'error_data' parameter to the FailAction 3f48e24 Implement notification of execution events 6683e15 Remove duplicate word 'the' af84269 Update the description about mistral client guide ce454f5 Hard code top package name 7423a58 Explain better combinations of task names and engine commands in docs 1f1463a Updated from global requirements 2de17cb Update reno for stable/queens 5410533 Updated from global requirements 1ae0827 Cache action definitions e8d6c38 Correction of comments for the #539039 review 8e64367 Add a step to install tox 79d1f87 Update mysql connection in doc 0ca36b2 Fix error sql about privileges in doc Diffstat (except docs and test files) ------------------------------------- .dockerignore | 73 ++ .gitignore | 2 + .stestr.conf | 3 + .testr.conf | 10 - .zuul.yaml | 49 +- README.rst | 234 +---- devstack/plugin.sh | 7 + devstack/settings | 4 + lower-constraints.txt | 178 ++++ mistral/actions/generator_factory.py | 6 +- mistral/actions/openstack/actions.py | 183 +++- mistral/actions/openstack/mapping.json | 304 +++++- mistral/actions/std_actions.py | 99 +- mistral/api/controllers/v2/action.py | 60 +- mistral/api/controllers/v2/action_execution.py | 8 +- mistral/api/controllers/v2/cron_trigger.py | 27 +- mistral/api/controllers/v2/environment.py | 10 +- mistral/api/controllers/v2/event_trigger.py | 38 +- mistral/api/controllers/v2/execution.py | 154 +-- mistral/api/controllers/v2/member.py | 36 +- mistral/api/controllers/v2/resources.py | 85 +- mistral/api/controllers/v2/task.py | 46 +- mistral/api/controllers/v2/validation.py | 2 +- mistral/api/controllers/v2/workbook.py | 74 +- mistral/api/controllers/v2/workflow.py | 44 +- mistral/auth/keycloak.py | 16 +- mistral/cmd/launch.py | 38 +- mistral/config.py | 175 +++- mistral/context.py | 4 +- .../versions/025_fix_length_task_name.py | 35 + .../versions/026_optimize_task_expression_func.py | 37 + .../027_add_last_heartbeat_to_action_execution.py | 51 + .../028_add_namespace_column_to_workbooks.py | 54 + .../versions/029_workbook_empty_namespace.py | 41 + mistral/db/sqlalchemy/migration/cli.py | 4 +- mistral/db/sqlalchemy/model_base.py | 5 +- mistral/db/utils.py | 23 +- mistral/db/v2/api.py | 87 +- mistral/db/v2/sqlalchemy/api.py | 333 ++++--- mistral/db/v2/sqlalchemy/filters.py | 9 + mistral/db/v2/sqlalchemy/models.py | 45 +- mistral/engine/action_queue.py | 3 +- mistral/engine/actions.py | 52 +- mistral/engine/base.py | 10 + mistral/engine/default_engine.py | 31 + mistral/engine/dispatcher.py | 61 +- mistral/engine/engine_server.py | 22 +- mistral/engine/policies.py | 20 +- mistral/engine/tasks.py | 102 +- mistral/engine/workflow_handler.py | 43 +- mistral/engine/workflows.py | 102 +- mistral/event_engine/default_event_engine.py | 3 +- mistral/exceptions.py | 57 +- mistral/executors/base.py | 3 - mistral/executors/default_executor.py | 6 +- mistral/executors/executor_server.py | 35 +- mistral/expressions/jinja_expression.py | 75 +- mistral/expressions/yaql_expression.py | 42 +- mistral/lang/base.py | 47 +- mistral/lang/parser.py | 46 +- mistral/lang/v2/actions.py | 4 +- mistral/lang/v2/on_clause.py | 87 +- mistral/lang/v2/policies.py | 6 +- mistral/lang/v2/publish.py | 4 +- mistral/lang/v2/retry_policy.py | 4 +- mistral/lang/v2/task_defaults.py | 34 +- mistral/lang/v2/tasks.py | 76 +- mistral/lang/v2/workbook.py | 4 +- mistral/lang/v2/workflows.py | 36 +- mistral/notifiers/__init__.py | 0 mistral/notifiers/base.py | 81 ++ mistral/notifiers/default_notifier.py | 44 + mistral/notifiers/notification_events.py | 82 ++ mistral/notifiers/notification_server.py | 93 ++ mistral/notifiers/publishers/__init__.py | 0 mistral/notifiers/publishers/noop.py | 31 + mistral/notifiers/publishers/webhook.py | 38 + mistral/notifiers/remote_notifier.py | 30 + mistral/policies/action.py | 15 + mistral/policies/workflow.py | 15 + mistral/rpc/base.py | 28 +- mistral/rpc/clients.py | 64 +- mistral/rpc/kombu/base.py | 2 +- mistral/rpc/kombu/kombu_client.py | 9 +- mistral/rpc/kombu/kombu_hosts.py | 32 +- mistral/rpc/kombu/kombu_server.py | 11 +- mistral/rpc/oslo/oslo_server.py | 2 +- mistral/serialization.py | 143 --- mistral/services/action_execution_checker.py | 83 ++ mistral/services/action_execution_reporter.py | 93 ++ mistral/services/periodic.py | 5 +- mistral/services/triggers.py | 3 + mistral/services/workbooks.py | 35 +- mistral/services/workflows.py | 15 +- ...rors-for-std-mistral-http-b852b6d8f0034477.yaml | 6 + .../actions/openstack/test_openstack_actions.py | 75 ++ .../unit/actions/test_std_mistral_http_action.py | 124 +++ .../unit/actions/test_std_test_dict_action.py | 32 + .../unit/engine/test_direct_workflow_rerun.py | 52 +- .../test_execution_fields_size_limitation.py | 46 +- .../unit/engine/test_reverse_workflow_rerun.py | 2 - .../unit/expressions/test_jinja_expression.py | 58 +- mistral/utils/__init__.py | 69 +- mistral/utils/expression_utils.py | 11 +- mistral/utils/javascript.py | 54 +- mistral/utils/openstack/keystone.py | 15 +- mistral/utils/rest_utils.py | 13 +- mistral/workflow/commands.py | 109 +- mistral/workflow/data_flow.py | 85 +- mistral/workflow/direct_workflow.py | 37 +- mistral/workflow/lookup_utils.py | 56 +- mistral/workflow/states.py | 20 + playbooks/docker-buildimage/post.yaml | 2 +- playbooks/docker-buildimage/run.yaml | 32 +- playbooks/legacy/mistral-ha/run.yaml | 9 +- playbooks/rally/run.yaml | 12 - rally-jobs/task-mistral.yaml | 2 +- ...ution-event-notifications-0f77c1c3eb1d6929.yaml | 7 + ...dd-missing-tacker-actions-dddcf77ddd90192f.yaml | 11 + .../add-publicize-policy-d3b44590286c7fdd.yaml | 7 + ...acer-javascript-evaluator-9d8f9e0e36504d72.yaml | 6 + ...task_execution_id-indexes-16edc58085e47663.yaml | 5 + ...action_definition_caching-78d4446d61c6d739.yaml | 9 + ...ion_for_oslo_rpc_executor-44afe1f728afdcb2.yaml | 17 + .../add_yaql_engine_options-200fdcfda04683ca.yaml | 12 + ...running-action-executions-b67deda65d117cee.yaml | 5 + ...jinja-expression-handling-135451645d7a4e6f.yaml | 5 + ...x-regression-when-logging-58faa35f02cefb34.yaml | 6 + .../notes/fix_pause_command-58294f613488511c.yaml | 9 + .../force-stop-executions-00cd67dbbc9b5483.yaml | 9 + .../improve_std_html_action-eca10df5bf934be8.yaml | 4 + .../load-keystoneauth-option-d9657d3052e82125.yaml | 11 + .../mistral-vitrage-actions-a205b8ea82b43cab.yaml | 3 + releasenotes/notes/namespace_for_workbooks.yaml | 15 + ...nt_data_from_task_context-c5281a5f5ae688f1.yaml | 10 + ...fe-rerun-in-task-defaults-87a4cbe12558bc6d.yaml | 4 + .../support-manila-action-8af256d5fadd1ac5.yaml | 4 + .../support-qinling-action-99cd323d4df36d48.yaml | 4 + .../notes/support-zun-action-3263350334d1d34f.yaml | 4 + ...ive_deletes_in_sqlalchemy-4b3006b3aba55155.yaml | 9 + ...t_evaluation_with_batches-6292ab64c131dfcc.yaml | 12 + ...environment_optimizations-deb8868df3f0dc36.yaml | 16 + releasenotes/source/index.rst | 1 + releasenotes/source/queens.rst | 6 + requirements.txt | 35 +- setup.cfg | 23 +- test-requirements.txt | 16 +- tools/cover.sh | 4 +- tools/docker/DOCKER_README.rst | 193 +++- tools/docker/Dockerfile | 81 +- tools/docker/build.sh | 9 - tools/docker/docker-compose/auth.json | 8 + tools/docker/docker-compose/infrastructure.yaml | 79 ++ .../docker/docker-compose/mistral-multi-node.yaml | 96 ++ .../docker/docker-compose/mistral-single-node.yaml | 22 + tools/docker/docker-compose/mistral.env | 4 + tools/docker/start.sh | 41 + tools/docker/start_mistral_rabbit_mysql.sh | 111 --- tools/get_action_list.py | 12 + tox.ini | 35 +- 236 files changed, 9706 insertions(+), 2426 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 7beb073..7fbac3f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10 +10 @@ cachetools>=2.0.0 # MIT License -eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT +eventlet!=0.20.1,>=0.20.0 # MIT @@ -12 +12 @@ gnocchiclient>=3.3.1 # Apache-2.0 -Jinja2!=2.9.0,!=2.9.1,!=2.9.2,!=2.9.3,!=2.9.4,>=2.8 # BSD License (3 clause) +Jinja2>=2.10 # BSD License (3 clause) @@ -15,5 +15,5 @@ keystonemiddleware>=4.17.0 # Apache-2.0 -mistral-lib>=0.3.0 # Apache-2.0 -networkx<2.0,>=1.10 # BSD -oslo.concurrency>=3.25.0 # Apache-2.0 -oslo.config>=5.1.0 # Apache-2.0 -oslo.context>=2.19.2 # Apache-2.0 +mistral-lib>=0.4.0 # Apache-2.0 +networkx>=1.10 # BSD +oslo.concurrency>=3.26.0 # Apache-2.0 +oslo.config>=5.2.0 # Apache-2.0 +oslo.context>=2.20.0 # Apache-2.0 @@ -32,2 +32,2 @@ pbr!=2.1.0,>=2.0.0 # Apache-2.0 -pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD -python-barbicanclient!=4.5.0,!=4.5.1,>=4.0.0 # Apache-2.0 +pecan>=1.2.1 # BSD +python-barbicanclient>=4.5.2 # Apache-2.0 @@ -40 +40,2 @@ python-keystoneclient>=3.8.0 # Apache-2.0 -python-mistralclient>=3.1.0 # Apache-2.0 +python-mistralclient!=3.2.0,>=3.1.0 # Apache-2.0 +python-manilaclient>=1.23.0 # Apache-2.0 @@ -43 +44 @@ python-muranoclient>=0.8.2 # Apache-2.0 -python-neutronclient>=6.3.0 # Apache-2.0 +python-neutronclient>=6.7.0 # Apache-2.0 @@ -49 +50 @@ python-troveclient>=2.2.0 # Apache-2.0 -python-ironicclient>=2.2.0 # Apache-2.0 +python-ironicclient>=2.3.0 # Apache-2.0 @@ -50,0 +52 @@ python-ironic-inspector-client>=1.5.0 # Apache-2.0 +python-vitrageclient>=2.0.0 # Apache-2.0 @@ -51,0 +54,2 @@ python-zaqarclient>=1.0.0 # Apache-2.0 +python-zunclient>=1.0.0 # Apache-2.0 +python-qinlingclient>=1.0.0 # Apache-2.0 @@ -53 +57 @@ PyJWT>=1.0.1 # MIT -PyYAML>=3.10 # MIT +PyYAML>=3.12 # MIT @@ -55,2 +59 @@ requests>=2.14.2 # Apache-2.0 -tenacity>=3.2.1 # Apache-2.0 -setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,>=16.0 # PSF/ZPL +tenacity>=4.4.0 # Apache-2.0 @@ -58 +61 @@ six>=1.10.0 # MIT -SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT +SQLAlchemy>=1.2.5 # MIT diff --git a/test-requirements.txt b/test-requirements.txt index 4b44a70..b8f5540 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -9 +8,0 @@ doc8>=0.6.0 # Apache-2.0 -eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT @@ -12 +11 @@ keystonemiddleware>=4.17.0 # Apache-2.0 -mistral-lib>=0.3.0 # Apache-2.0 +mistral-lib>=0.4.0 # Apache-2.0 @@ -14 +13 @@ mock>=2.0.0 # BSD -networkx<2.0,>=1.10 # BSD +networkx>=1.10 # BSD @@ -22,7 +21,2 @@ os-api-ref>=1.4.0 # Apache-2.0 -pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD -reno>=2.5.0 # Apache-2.0 -requests-mock>=1.1.0 # Apache-2.0 -sphinx!=1.6.6,>=1.6.2 # BSD -sphinxcontrib-httpdomain>=1.3.0 # BSD -sphinxcontrib-pecanwsme>=0.8.0 # Apache-2.0 -openstackdocstheme>=1.18.1 # Apache-2.0 +oauthlib>=0.6.2 # BSD +requests-mock>=1.2.0 # Apache-2.0 @@ -31 +25 @@ tempest>=17.1.0 # Apache-2.0 -testrepository>=0.0.18 # Apache-2.0/BSD +stestr>=2.0.0 # Apache-2.0