We are pumped to announce the release of: watcher 13.1.0 This release is part of the dalmatian release series. The source is available from: https://opendev.org/openstack/watcher Download the package from: https://tarballs.openstack.org/watcher/ Please report issues through: https://bugs.launchpad.net/watcher/+bugs For more details, please see below. 13.1.0 ^^^^^^ Security Issues *************** * Watchers no longer forges requests on behalf of a tenant when swapping volumes. Prior to this release watcher had 2 implementations of moving a volume, it could use cinders volume migrate api or its own internal implementation that directly calls nova volume attachment update api. The former is safe and the recommend way to move volumes between cinder storage backend the internal implementation was insecure, fragile due to a lack of error handling and capable of deleting user data. Insecure: the internal volume migration operation created a new keystone user with a weak name and password and added it to the tenants project with the admin role. It then used that user to forge request on behalf of the tenant with admin right to swap the volume. if the applier was restarted during the execution of this operation it would never be cleaned up. Fragile: the error handling was minimal, the swap volume api is async so watcher has to poll for completion, there was no support to resume that if interrupted of the time out was exceeded. Data-loss: while the internal polling logic returned success or failure watcher did not check the result, once the function returned it unconditionally deleted the source volume. For larger volumes this could result in irretrievable data loss. Finally if a volume was swapped using the internal workflow it put the nova instance in an out of sync state. If the VM was live migrated after the swap volume completed successfully prior to a hard reboot then the migration would fail or succeed and break tenant isolation. see: https://bugs.launchpad.net/nova/+bug/2112187 for details. Bug Fixes ********* * All code related to creating keystone user and granting roles has been removed. The internal swap volume implementation has been removed and replaced by cinders volume migrate api. Note as part of this change Watcher will no longer attempt volume migrations or retypes if the instance is in the *Verify Resize* task state. This resolves several issues related to volume migration in the zone migration and Storage capacity balance strategies. While efforts have been made to maintain backward compatibility these changes are required to address a security weakness in watcher's prior approach. see: https://bugs.launchpad.net/nova/+bug/2112187 for more context. Changes in watcher 13.0.0..13.1.0 --------------------------------- b5b1bc54 Configure watcher tempest's microversion in devstack c049a533 use cinder migrate for swap volume dbc06d15 Skip real-data tests in non-real-data jobs 54b3b584 Further database refactoring 8b0f1dbf Replace deprecated LegacyEngineFacade 238bb50f Run watcher-db-manage in grenade testing from venv db85f326 Update TOX_CONSTRAINTS_FILE for stable/2024.2 f6015fd6 Update .gitreview for stable/2024.2 Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + .zuul.yaml | 1 + devstack/lib/watcher | 16 ++ devstack/plugin.sh | 3 + devstack/upgrade/upgrade.sh | 2 +- .../notes/bug-2112187-763bae283e0b736d.yaml | 47 +++++ tox.ini | 2 +- watcher/applier/actions/volume_migration.py | 98 +++-------- watcher/common/context.py | 3 + watcher/common/keystone_helper.py | 34 ---- watcher/common/utils.py | 8 +- watcher/db/sqlalchemy/alembic/env.py | 4 +- watcher/db/sqlalchemy/api.py | 195 ++++++++++----------- watcher/db/sqlalchemy/migration.py | 6 +- watcher/db/sqlalchemy/models.py | 8 - watcher/decision_engine/audit/continuous.py | 2 +- .../strategy/strategies/zone_migration.py | 55 +++--- .../decision_engine/audit/test_audit_handlers.py | 14 +- .../strategy/strategies/test_zone_migration.py | 5 +- 21 files changed, 274 insertions(+), 323 deletions(-)