Open Stack

We are gleeful to announce the release of:

mistral 7.1.0: Mistral Project

This release is part of the rocky stable release series.

The source is available from:

    https://opendev.org/openstack/mistral

Download the package from:

    https://pypi.org/project/mistral

Please report issues through:

    https://bugs.launchpad.net/mistral/+bugs

For more details, please see below.

7.1.0
^^^^^


New Features
************

* This makes getting a root_execution_id available to the jinja
  execution object. Before this it was only possible to get that
  through filtering and querying the executions search.

* Added HTTPProxyToWSGI middleware in front of the Mistral API. The
  purpose of this middleware is to set up the request URL correctly in
  the case there is a proxy (for instance, a loadbalancer such as
  HAProxy) in front of the Mistral API. The HTTPProxyToWSGI is off by
  default and needs to be enabled via a configuration value. Fixes
  [bug 1590608 (https://bugs.launchpad.net/mistral/+bug/1590608)]
  Fixes [bug 1816364
  (https://bugs.launchpad.net/mistral/+bug/1816364)]

* It's now possible to add reply-to address when sending email.


Bug Fixes
*********

* Mistral doesn't log enough info about sending actions to executor
  and receiving them on the executor side. It makes it hard to debug
  situations when an action got stuck in RUNNING state. It has now
  been fixed by adding additional log statements.

* Added the "convert_input_data" config property under the "yaql"
  group. By default it's set to True which preserves the current
  behavior so there's no risk with compatibility. If set to False, it
  disables the additional data conversion that was initially added to
  support some tricky cases like working with sets of dicts (although
  dict is not a hashable type and can't be put into a set). Disabling
  it give a significant performance boost in cases when data contexts
  are very large.

* Fixed a backward compatibility issue: there was a change made in
  Rocky that disallowed the 'params' property of a workflow execution
  to be None when one wants to start a workflow.

* [bug 1837468 (https://bugs.launchpad.net/keystone/+bug/1837468)]
  Fixed
  unit.actions.openstack.test_generator.GeneratorTest.test_generator
  failure by avoid triggering version discovery as it needs to talk to
  Keystone server. Talking to a live Keystone server is not needed for
  this particular unit test.

* Sometimes Mistral was raising DetachedInstanceError for action
  defintions coming from cache. It's now fixed by cloning objects
  before caching them.

* [bug 1715848 (https://bugs.launchpad.net/mistral/+bug/1715848)]
  Fixed a bug that prevents event-engines to work correctly in HA.

* Fix error validate token when run cron trigger. The problem is
  that a trust client can't do validate token when run cron trigger.

* Fixed the issue when "join" task remained in WAITING state forever
  if the last inbound task failed and it was not a direct predecessor.

* If an action execution fails but returns a result as a list
  (error=[]) the result of this action is assigned to the task
  execution 'state_info' field which is a string according to the DB
  model. On Python 3 it this list magically converts to a string. On
  Python 2.7 it doesn't. The reason is probably in how SQLAlchemy
  works on different versions of Python. This has now been fixed with
  an explicit type coercion.

* Workflow execution integrity checker mechanism was too agressive
  in case of big workflows that have many task executions in RUNNING
  state at the same time. The mechanism was selecting them all in one
  query and calling "on_action_complete" for each of them within a
  single DB transaction. That could lead to situations when this
  mechanism would totally block all normal workflow processing whereas
  it should only be a "last chance" aid in case of real infrastructure
  failures (e.g. MQ outage). This issue has been fixed by adding a
  configurable batch size, so that the checker can't select more than
  this number of task executions in RUNNING state at once.

* For an ad-hoc action, preparing input for its base action was done
  more than once. It happened during the validation phase and the
  scheduling phase. However, input preparation may be expensive in
  case of heavy expressions and data contexts. This has now been fixed
  by caching a prepared input within an AdHocAction instance.

* Action heartbeat checker was using scheduler to process expired
  action executions periodically. The side effect was that upon system
  reboot there may have been duplicating delayed calls in the
  database. So over time, the number of such calls could be
  significant and those jobs could even affect performance. This has
  now been fixed with regular threads without using scheduler at all.
  Additionally, the new configuration property "batch_size" has been
  added under the group "action_heartbeat" to control the maximum
  number of action executions processed during one iteration of the
  action execution heartbeat checker.

* Action execution checker didn't set a security context before
  failing expired action executions. It caused
  ApplicationContextNotFoundException in case if corresponding
  workflow specification was not in the cache and Mistral had to load
  a DB object. The DB operation in turn was trying to access a
  security context which wasn't set. It's now fixed by setting an
  admin context in the action execution checker thread.

* Workflow and join completion check logic is now simplified with
  using post transactional queue of operations which is a more generic
  version of action_queue module previously serving for scheduling
  action runs outside of the main DB transaction. Workflow completion
  check is now registered only once when a task completes which
  reduces clutter and it's registered only if the task may potentially
  lead to workflow completion.

* WorkflowExecution database model had only "root_execution_id" to
  reference a root workflow execution, i.e. the most parent workflow
  execution in the execution tree. So if we needed to get an entity
  itself we'd always make a direct query to the database, in fact, w/o
  using an entity cache in the SQLAlchemy session. It's now been fixed
  by adding a normal mapped entity for root workflow execution. In
  other words, WorkflowExecution class now has the property
  "root_execution". It slightly improves performance in case this
  property is accessed more than once per the database session.

Changes in mistral 7.0.4..7.1.0
-------------------------------

4d3ef888 Mask sensitive data when logging action results
4dc6cda9 Make the mistral devstack jobs non-voting in Rocky
eab767f3 Add release note for fix error of cron trigger run
d1b8dd16 Fix error validate token when run cron trigger
f2a0b569 disable triggering version discovery in test_generator unit tests
c9e61b80 Add release notes to changes to wflanguage execution object
ff1e5b6e Process all task batches in wf output evaluation
81daaf09 Adding root_execution_id to the jinja executor function
9e128262 OpenDev Migration Patch
b1b888e7 Add release notes for reply-to feature
5be18b7e Add reply-to to std.email
e23b5ced Add release note for I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36
6e05155c Add http_proxy_to_wsgi middleware
c8865a33 Add "convert_input_data" config property for YAQL expressions
cddaeda9 Add release notes for engine optimizations
77a68312 Add "root_execution" mapped property to WorkflowExecution model
192af4ee Optimize action scheduling
35dc4eec Replace openstack.org git:// URLs with https://
fa0c4f12 Release note for fixing event-engines HA
ec06ccf6 [Event-engine] Allow event_engine to work in HA
ce3d8607 Add loging for sending an action to executor
5e5901bb Set admin security context in the action execution checker thread
a3059ba7 Remove tripleo newton and ocata jobs (rocky branch)
30822d57 Fix "join" when the last indirect inbound task failed
dc3a2a81 Fix race condition in refreshing "join" task state
9b302e2d Clone cached action definitions
8133811a Refactor action execution checker without using scheduler
d8b19e93 Add batch size for integrity checker
6774a23b Simplify workflow and join completion logic
aaebbe40 An execution hangs in the RUNNING state after rerun
4c911a76 Fix how action result is assigned to task 'state_info' field
5d128805 Allow None for 'params' when starting a workflow execution
21299860 Improve workflow completion logic by removing periodic jobs


Diffstat (except docs and test files)
-------------------------------------

.gitreview                                         |   2 +-
.zuul.yaml                                         |  25 ++-
bindep.txt                                         |   8 +
mistral/actions/std_actions.py                     |  18 +-
mistral/api/app.py                                 |   4 +
mistral/api/controllers/root.py                    |   2 +-
mistral/api/controllers/v2/execution.py            |   2 +-
mistral/api/controllers/v2/root.py                 |   3 +-
mistral/config.py                                  |  41 ++++-
mistral/context.py                                 |   2 +-
mistral/db/v2/api.py                               |  12 +-
mistral/db/v2/sqlalchemy/api.py                    |  19 ++-
mistral/db/v2/sqlalchemy/models.py                 |   7 +
mistral/engine/action_queue.py                     | 133 ---------------
mistral/engine/actions.py                          |  85 ++++++----
mistral/engine/default_engine.py                   |  32 ++--
mistral/engine/engine_server.py                    |   5 +-
mistral/engine/policies.py                         |  22 +--
mistral/engine/post_tx_queue.py                    | 131 +++++++++++++++
mistral/engine/task_handler.py                     |  55 +++++-
mistral/engine/tasks.py                            |  59 ++++++-
mistral/engine/workflow_handler.py                 | 185 +++++++++------------
mistral/engine/workflows.py                        |  85 ++++++----
mistral/event_engine/default_event_engine.py       |   2 -
mistral/executors/executor_server.py               |  13 +-
mistral/rpc/base.py                                |   2 +-
mistral/rpc/clients.py                             |  18 +-
mistral/rpc/kombu/kombu_client.py                  |   2 +-
mistral/rpc/oslo/oslo_client.py                    |  10 +-
mistral/services/action_execution_checker.py       |  99 +++++++----
mistral/services/triggers.py                       |   2 +
.../unit/engine/test_direct_workflow_rerun.py      |  71 ++++++++
.../engine/test_direct_workflow_with_cycles.py     |  43 +++--
.../unit/expressions/test_jinja_expression.py      |   8 +-
mistral/utils/expression_utils.py                  |  10 +-
mistral/utils/openstack/keystone.py                |   8 +-
mistral/utils/rest_utils.py                        |   2 +-
mistral/workflow/base.py                           |   5 +
mistral/workflow/data_flow.py                      |   5 +-
mistral/workflow/direct_workflow.py                |  15 +-
mistral/workflow/lookup_utils.py                   |  14 +-
playbooks/legacy/mistral-ha/run.yaml               |   2 +-
...gging_for_sending_actions-c2ddd97027803ecd.yaml |   7 +
...oot_execution_id_to_jinja-90b67c69a50370b5.yaml |   5 +
...nput_data_config_property-09822dee1f46eb8e.yaml |  11 ++
...workflow_execution_params-f25b752e207d51d7.yaml |   6 +
.../notes/bug-1837468-94a8a2e4326ab1e5.yaml        |   8 +
...cached_action_definitions-e8b6005b467f35f2.yaml |   5 +
.../fix-event-engines-ha-cc78f341095cdabf.yaml     |   5 +
...ken_when_run_cron_trigger-7beffc06b75294fb.yaml |   5 +
...t_finished_indirect_error-b0e5adf99cde9a58.yaml |   5 +
...ask_state_info_assignment-e25481ce8c3193ba.yaml |   9 +
...y-to-wsgi-oslo-middleware-f66f1b9533ea1e8a.yaml |  11 ++
...checker_work_with_batches-56c1cd94200d4c38.yaml |  13 ++
..._adhoc_actions_scheduling-e324f66f962ae409.yaml |   8 +
...rtbeats_without_scheduler-9c3500d6a2b25a4d.yaml |  12 ++
..._action_execution_checker-eee7fb697fb213d1.yaml |   9 +
...and_join_completion_check-77a47c5d8953096d.yaml |  10 ++
.../notes/std.email-reply-to-c283770c798db7d0.yaml |   3 +
...entity_for_root_execution-1af6af12ee437282.yaml |  11 ++
tools/config/config-generator.mistral.conf         |   1 +
76 files changed, 1397 insertions(+), 480 deletions(-)