We are satisfied to announce the release of: cinder 19.0.0: OpenStack Block Storage This release is part of the xena release series. The source is available from: https://opendev.org/openstack/cinder Download the package from: https://tarballs.openstack.org/cinder/ Please report issues through: https://bugs.launchpad.net/cinder/+bugs For more details, please see below. 19.0.0 ^^^^^^ Known Issues ************ * It is currently possible to manage a volume to an encrypted volume type, but that is not recommended because there is no way to supply an encryption key for the volume to cinder. Un-managing a volume of an encrypted volume type is already prevented, and it is expected that management to an encrypted type will similarly be blocked in a future release. This issue is being tracked as Bug #1944577 (https://bugs.launchpad.net/cinder/+bug/1944577). * **Cinder use of cgroups v1** This note applies to deployments that are using the cinder configuration option "volume_copy_bps_limit" in its non-default value (the default is 0). The cinder-volume service currently depends on Linux Kernel Control Groups (cgroups) version 1 (https://www.kernel.org/doc/html/latest /admin-guide/cgroup-v1/cgroups.html) to control i/o throttling during some volume-copy and image-convert operations. Some Linux distributions, however, have changed to using cgroup v2 (https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) by default and may have discontinued cgroups v1 support completely. Consult your Linux distribution's documentation for details. The cinder team is working on a throttling solution using cgroup v2, but it was not ready at the time of this release. The solution is expected to be backported to a future release in the Xena series. This issue is being tracked as Bug #1942203 (https://bugs.launchpad.net/cinder/+bug/1942203). * There is a race condition between the delete attachment and delete volume operations that has been observed when running cinder-csi. This race can leave deleted volumes stuck as attached to instances. The cinder team is working on a solution which is expected to be backported to a future release in the Xena series. The issue is being tracked as Bug #1937084 (https://bugs.launchpad.net/cinder/+bug/1937084). * When the Ceph backup driver is used for the backup service, restoring a backup to a volume created on a non-RBD backend fails. The cinder team is working on a solution which is expected to be backported to a future release in the Xena series. The issue is being tracked as Bug #1895035 (https://bugs.launchpad.net/cinder/+bug/1895035). * Creating a volume of an encrypted volume type from an image in the Image service (Glance) using the generic NFS driver results in an unusable volume. The cinder team is working on a solution which is expected to be backported to a future release in the Xena series. The issue is being tracked as Bug #1888680 (https://bugs.launchpad.net/cinder/+bug/1888680). * **NFS-based backend drivers and qcow2 version 2 images** Under some circumstances, NFS-based backend drivers will store a volume as a qcow2 image. Thus cinder allows for the possibility that an operator may choose to manage a storage object in an NFS- based backend that is a qcow2 image. Version 3 of the qcow2 format has been the default for qcow2 creation in qemu-img since QEMU-1.7 (December 2013), and operating system vendors are discussing discontinuing (or limiting) support of the version 2 format in upcoming releases. Thus, we recommend that operators who want to manage a storage object in an NFS-based storage backend as a cinder volume should not do this with a qcow2 image that is in the version 2 format, but should change it to the qcow2-v3 format first. Note: The format version of a qcow2 can be determined by looking for the "compat" field in the output of the "qemu-img info" command. A version 2 format image will report "compat=0.10", whereas a qcow2 in version 3 format will report "compat=1.1". Upgrade Notes ************* * **RBD driver: Enable Ceph V2 Clone API and Ceph Trash auto purge** In light of the fix for RBD driver bug #1941815 (https://bugs.launchpad.net/cinder/+bug/1941815), we want to bring the following information to your attention. Using the v2 clone format for cloned volumes allows volumes with dependent images to be moved to the trash - where they remain until purged - and allow the RBD driver to postpone the deletion until the volume has no dependent images. Configuring the trash purge is recommended to avoid wasting space with these trashed volumes. Since the Ceph Octopus release, the trash can be configured to automatically purge on a defined schedule. See the "rbd trash purge schedule" commands in the rbd manpage (https://docs.ceph.com/en/octopus/man/8/rbd/). Bug Fixes ********* * RBD driver bug #1941815 (https://bugs.launchpad.net/cinder/+bug/1941815): Fixed deleting volumes with snapshots/volumes in the ceph trash space. Changes in cinder 18.0.0.0rc1..19.0.0 ------------------------------------- 1eb7384cb Use os-brick 5.0.1 93fb8fbe1 Add "known issues" to xena release note aa0a324cc Add warning about qcow2 version 2 format images faada70a0 Fix cinder-manage db version e1fb66278 fixtures: Don't persist state in the Database fixture 31a237b72 Fix typo in release note file d362c1e78 Raise min of oslo.db to 11.0.0 5226a13d1 RBD: Call trash operation when plain deletion fails 7542e1af9 Specify xena os-brick series in requirements 8226a703a Add deprecation notices to deprecated policy rules f0b486d86 Update SQLAlchemy related requirements abd7997fb Raise min of oslo.policy to 3.8.1 63dff676c Raise minimum oslo.log to 4.5.0 5a5e2276a Update TOX_CONSTRAINTS_FILE for stable/xena 7e059db63 Update .gitreview for stable/xena 72cba373d Correct "Add release note about xena policy changes" cfa3329fc Correct "Clean up policy matrix for Xena release" 122457449 mypy: Fix unused type: ignore in manager.py b1dc74d2e Add release note about xena policy changes 850d657ad Clean up policy matrix for Xena release 0700979bb Add release note prelude for the Xena release c806c74c8 Update policies related to user visible extra specs 88b993b19 Implement project personas for snapshot metadata f91c4ba82 Implement Xena project personas for group snapshots 64ba8e08a Implement project personas for backups API 639224f90 Implement project personas for volume actions e0335de8b Better workaround for policy file in PDF docs 58b193887 Remove sample policy config from docs generation bdcee8523 [SVF] Fix extend issue for mirroring volumes c217dd446 Implement Xena project personas for group_actions 640c7302d Add request method to test requests 91482ac12 [SVF]:Storwize HyperSwap snapshot clone is failing 7ff8bb89a Implement Xena personas for volume groups 9843ac3bb Implement Xena project personas for volume types 5fc7df24a Xena project personas for volume type access API 85d64c647 Implement project personas for snapshots 8ebe3ecef Implement secure RBAC for snapshot actions 4073c1abc Implement project personas for volume metadata API 68ef97cd4 Implement project personas for the limits API 53c3b192e Fix: Online migration for volume_use_quota_online_data_migration d6696f875 Implement project personas group_types 1bb851660 Update project personas for default_types 37377838d Implement project personas for volume API e0f4ab228 Implement project personas for quotas and quota_classes 1c93a9a36 Implement project personas for volume transfers 1c0c25bab Native multibackend-matrix Zuul v3 job 8a240585b [SVF]:HyperSwap volume service status update 36e008757 Hitachi: Add generic volume groups 40374deac Nimble: Add Alletra 6k related information a6488ba87 [SVF] Update rccg details for mirror volumes 3baa9ad09 JovianDSS: add multiattach and 16K block support 8930fbac3 JovianDSS: fix code style and naming 6906cd2a7 NetApp ONTAP: Add option to report storage provisioned capacity 1f96b386b Netapp ONTAP: Add support to revert to snapshot 35a073010 Clean up user visible extra specs release note 084212711 Implement user visible extra specs ea50df5d2 JovianDSS: fix iscsi target recovery function d3e8af04a HPE 3PAR: Add HPE Alletra 9k related information 8c7ca26fb Remove extraneous space from error message 6c6c18589 Docs: Discourage using naked rechecks 2a017f5b3 NetApp ONTAP: Add storage assisted migration support 3cedf7d81 Implement project personas for attachments 810733d74 Implement project personas for messages 81e0da35d Add infrastructure for testing new RBAC policies 9ed7c16ef Snapshot in-use volumes without force flag dab252f1f Update mv version history document for 3.65 e733bdb48 [SVF] Manage GMCV volumes on separate pools 7fb4f272c Update base policy file 400008194 Correct the rest api url of import_record of backup 4b246564a db: Remove 'db' argument from various managers 8353d6e20 db: Remove 'db_driver' option ee771bbdb db: Integrate alembic 069b4c477 db: Add all migrations to initial alembic migration a38b3546c db: Add initial alembic migration 51498ffe1 db: Add alembic boilerplate 178272944 db: Vendor 'oslo_db.sqlalchemy.migration' e823386cb db: Move sqlalchemy-migrate migrations 3b7f49986 Add user messages for backup operations 269d21b2e Test glance with cinder backend optimization ec44fc899 Expose volume & snapshot use_quota field 94dfad99c Improve quota usage for temporary resources b78997c2b Clear OVO history and compatibility 28d9bca7d Fix notifications of migration temp volume c995438ca Change 'host' option from HostAddressOpt to StrOpt d29f12117 Update snapshot api microversion docs and tests 0dc4daa21 [SVF]:Fix multiple lshost calls during attach. 4a5ff4eb7 Support Images API v2.11 6a0b41a8f Log connection info returned from driver f882b0c65 [SVF]:Fix add volumes to GMCV group 91c7f040a [SVF]:Retype in-use hyperswap volume da172417f [SVF] Fixed update_rep_properties for empty values c4b895676 LVM: Retry lvextend commands on code 139 70a07ee7a [SVF]:Fix create volume on drp f179df086 Add releasenote for schema validation fix 3753aa1b6 Doc: IBM Storages user documentation update eaafd81e9 [SVF] Fix issue to get volume relationship details b5ac2af0c mypy: continued manager, scheduler, rpcapi d613ce06c conditional_update(): disable false pylint error message 8c46c09ad mypy: image cache b10f71e42 mypy: create_volume flows d4c112de2 Fix cinder-manage clean_locks command 672a0e979 PowerMax Driver - Fix for create snapshot ad0a9eebf [SVF]: Fix Enable Replication for Storwize_V5000E 7b9143fae Doc: Remove backup_id from backup import url and parameter e429fa974 Add cinder permissions matrix 81831c915 Remove 'enable_v3_api' option dbcbfefe5 Add cinder-manage command to remove file locks d2f6ec556 Remove file locks once we delete a resource f67eae51f [Pure Storage] Resolve missing provider_id issue (PowerVC) 49a2c85ed RBD: use correct stripe unit in clone operation cf1c52529 [rbd] Fix create encrypted volume from snapshot 5714bdb84 Allow removing NFS snapshots in error status bd0e409b7 Replace deprecated tenacity.Retrying.call 685426508 Add openstacksdk-functional-devstack job in cinder gate 98de5e08b Doc: api-ref doc update eb5c8183b [Pure Storage] Fix CG cloning crash when very long volume names cc6b499d2 Run cinder-mypy job in the check c8c4fc953 Add installation of mypy stubs packages baabc7bd8 db: Reorder initial migration 2b2993cdd db: Use 'import sqlalchemy as sa' pattern 4f55f05a9 Tests: Fix InvalidUUID warnings b604a3bf6 Drop lower-constraints jobs 25eb0a7d7 NFS: Update connection info on online snap create 334485dc5 mypy: image_utils d0f317ab4 LVM: Simplify version fetching 5c0889447 Tests: Simplify LVM fake_execute 49848a767 [Pure Storage] Fix minimum SDK version required b29223840 Add backup_swift_create_storage_policy config opt a8552ed2d LVM: Retry lvdisplay and lvcreate calls on segfault 6022cea4a Use mv.LIMITS_ADMIN_FILTER in the code 2da7d42b9 Retry "lvs" call on segfault for _get_thin_pool_free_space 9357026a7 streamline _report_driver_status method 560318c82 Fix: Schema validation for attachment create API 02f609c71 LVM: Use --readonly for lvdisplay in lv_has_snapshot 8c5de2cfd Fix typo in Dell EMC Unity driver documentation 74ac9090b Update decode_cipher doc text about encryption ciphers ca1e0bd76 Add libcgroup related packages in bindep.txt 1f65c2a92 PowerMax Driver - Allow for case mismatch in SGs 7cb04d4f4 Move require_driver_initialized / log_unsupp to volume_utils da100e1fc NetApp ONTAP: Fix QoS lost after moving volume dbf78b8c2 Update Block Storage API v2 api-ref e05b261af Remove Block Storage API v2 e832ddb06 [SVF] Support volume-resize to hyperswap volume 9cd128f07 [SVF] Fixed add volumes to clone group issue 0c5511f84 [Pure Storage] Ensure multiattach volumes are not disconnected early 020944bad [SVF]: Fix extend issue for a clone of rep-volume e6ad64404 [SVF]:Bulk create Hyperswap volume is failing. dab5df174 Update volume api microversion doc and tests 2e42cd743 PowerMax Docs - Corrections to QoS section a0eed589a Update IRC info da2fbff24 Fix functional jobs 9d5b9728f Doc: Remove incorrect note about encryption flag 8d84bdab6 Tests: Don't assert notifier not called 9af47a2fc PowerMax Driver - Fix for create group from source a449cda21 Replace getargspec with getfullargspec a3682eac3 SQLA 1.4: Fix calculate_resource_count 55046123f SQLA 1.4: Fix conditional update Case e8c791045 api-ref: Use key_size 256 in API examples 0a1cc1a2d Abort volume creation when encryption spec is invalid 0367972a3 remove the oslo_utils.fnmatch 87b8c88dc Tests: Fix oslo.i18n warning 2c2370671 zuul: fixes for the A/A job (nodeset, variables) cded69efe Fix PowerFlex volume type conversion b841a3aff volume api: Remove unused get_snapshot_metadata_value method 895334082 mypy: annotate volume_utils / utils / exc f13717d12 zuul: add mypy experimental job 23dd35d68 Drop support for SQLite < 3.7 290033ff3 image_utils: Remove unused _validate_file_format method b64fef783 filter_scheduler: Remove schedule() method ce465cb62 Remove references to 'sys.version_info' 734012c52 vmware: Use oslo.vmware's get_moref_value() 0541efb67 vmware tests: Support different moref backend representations fd6ce6306 PowerMax Docs - Wallaby documentation b29bb44e9 db: Compact migrations to Train 59c5f1e7a db: Remove 'cinder.database.migration_backend' entrypoint ea7ae8e65 setup.cfg: Replace dashes with underscores fb2dde068 Tests: Add coverage for coordinator start/stop ceb1c2288 Remove unused _get_non_shared_target_hosts from cmd/manage 738e77a0f Unit test RBD clone depth calculation 70912ef15 api-ref: Fix "name" parameter 75588809b Update release note info 9a0d7a789 Add Python3 xena unit tests 5edc77a18 Driver assisted migration on retype when it's safe 15d8ad15d pylint: Fix migration E1120 no-value-for-parameter 764bc3c5d pylint: Fix E0213 no-self-argument in tests d2d5e9250 pylint: Fix E1120 no-value-for-parameter for mocks 1c3cda154 Fix sporadic cleanup unit test failure 738f81fee Fix instance locality scheduler filter ecfa22588 Pure Storage - bump version numbers for Xena f97778189 Remove uuid check/generate in service 5d7e7547f Open local image files with "rb" mode b5b681b73 Bug fix for free space calculation 3475957b8 Reserve migrations for DB backports fe55090f9 PowerMax Driver - Fix for deleting replication group 91b133d82 Update to pylint 2.7.4 b675b66d4 Fix: show volume by name for non-admins dcf66a96a Modify/Move project validation methods to api_utils a166f7bb1 api-ref: add additional info for mv 3.64 57eb79d26 Remove unused _db_error code from cmd/manage.py a6c86b104 Set Wallaby maximum mv in REST API version history bcb33db88 Quota: Fix multiple race conditions 935ddf7c7 Prevent quota and reservations to go into negative cdd60605f Update cinder manage quota commands 1fb0767d8 Fix quota usage duplicate entries 6005fc25f Remove nested quota leftovers 85aee67dc Quota: Fix until_refresh config changes 29edc0fd2 Remove unused quota python classes b50a57137 Add quota utils to cinder-manage 5213b4cf3 Update help text for backup compression option 9c94ed3c6 Use os-brick 4.3.1 0b2c0d923 LVM: Fix delete volume error due to lvs failure b77fb72aa Change snapshots type with volume retype 8f03b26f5 Fix automatic quota sync for temporary volumes c29610569 Remove unnecessary DB read b9f525b7d Remove unused code path in attachment_delete 99c7f1dcd Update master for stable/wallaby e1f56a7c8 Cinder - Creating clone of encrypted volume fails 1a267bf6f PowerMax Driver - Temporary snapshot enhancements c98f88208 api-ref: Fix "id" parameter 004f01f9f Make availability zone type exception easier to understand. d6a407b43 Update contributor doc with additional meetings 5dea649c4 Backup manager: Synchronously call remove_export_snapshot f0abdefed Fix PowerStore iSCSI targets filtering 83ff2ae9e Remove six from quota.py 1db0e34cc Remove six from cinder.tests.unit.objects 7f84fff32 Fix invalid yaml in reno template 6d85d05f7 Add warning message about slow volume backend 7e382b478 [doc] remove outdated package python-keystone 7aa5e95d4 [PURE] support IPv6 / add parameter pure_iscsi_cidr_list 4946c787d Tests: test_backup - specify volume_type_id for snapshots e96f48d36 image_utils: Simplify fetch_verify_image 993bf354c Tests: Solidfire - Use UUIDs in object UUID fields 52997e56c HPMSA: Report SAN driver options 8f6a3cac3 Tests: Improve cascade delete coverage be2f4a5f5 Excess availability_zone judgment code Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + .pylintrc | 1 + .zuul.yaml | 91 +- api-ref/source/v2/ext-backups.inc | 3 +- api-ref/source/v2/index.rst | 21 +- .../v2/samples/encryption-type-create-request.json | 2 +- .../samples/encryption-type-create-response.json | 2 +- .../v2/samples/encryption-type-show-response.json | 2 +- api-ref/source/v3/attachments.inc | 2 +- api-ref/source/v3/ext-backups.inc | 3 +- api-ref/source/v3/os-vol-transfer-v3.inc | 10 +- api-ref/source/v3/parameters.yaml | 68 +- api-ref/source/v3/resource-filters.inc | 2 + .../snapshots/snapshot-create-response.json | 2 +- .../samples/snapshots/snapshot-show-response.json | 2 +- .../snapshots/snapshot-update-response.json | 2 +- .../snapshots-list-detailed-response.json | 2 +- .../snapshots/v3.14/snapshot-create-response.json | 16 + .../snapshots/v3.14/snapshot-show-response.json | 18 + .../snapshots/v3.14/snapshot-update-response.json | 16 + .../v3.14/snapshots-list-detailed-response.json | 20 + .../snapshots/v3.41/snapshot-create-response.json | 17 + .../snapshots/v3.41/snapshot-show-response.json | 19 + .../snapshots/v3.41/snapshot-update-response.json | 17 + .../v3.41/snapshots-list-detailed-response.json | 21 + .../snapshots/v3.65/snapshot-create-response.json | 18 + .../snapshots/v3.65/snapshot-show-response.json | 20 + .../snapshots/v3.65/snapshot-update-response.json | 18 + .../v3.65/snapshots-list-detailed-response.json | 22 + .../v3/samples/versions/version-show-response.json | 4 +- .../v3/samples/versions/versions-response.json | 28 +- .../encryption-type-create-request.json | 2 +- .../encryption-type-create-response.json | 2 +- .../volume_type/encryption-type-show-response.json | 2 +- .../volumes/v3.13/volume-create-response.json | 35 + .../volumes/v3.13/volume-show-response.json | 39 + .../volumes/v3.13/volume-update-response.json | 37 + .../v3.13/volumes-list-detailed-response.json | 41 + .../volumes/v3.21/volume-create-response.json | 36 + .../volumes/v3.21/volume-show-response.json | 40 + .../volumes/v3.21/volume-update-response.json | 38 + .../v3.21/volumes-list-detailed-response.json | 42 + .../volumes/v3.48/volume-create-response.json | 38 + .../volumes/v3.48/volume-show-response.json | 42 + .../volumes/v3.48/volume-update-response.json | 40 + .../v3.48/volumes-list-detailed-response.json | 44 + .../volumes/v3.61/volume-create-response.json | 39 + .../volumes/v3.61/volume-show-response.json | 43 + .../volumes/v3.61/volume-update-response.json | 41 + .../v3.61/volumes-list-detailed-response.json | 45 + .../volumes/v3.63/volume-create-response.json | 40 + .../volumes/v3.63/volume-show-response.json | 44 + .../volumes/v3.63/volume-update-response.json | 42 + .../v3.63/volumes-list-detailed-response.json | 46 + .../volumes/v3.65/volume-create-response.json | 41 + .../volumes/v3.65/volume-show-response.json | 45 + .../volumes/v3.65/volume-update-response.json | 43 + .../v3.65/volumes-list-detailed-response.json | 47 + .../v3/samples/volumes/volume-create-response.json | 2 +- .../v3/samples/volumes/volume-show-response.json | 2 +- .../v3/samples/volumes/volume-update-response.json | 2 +- .../volumes/volumes-list-detailed-response.json | 10 +- api-ref/source/v3/vol-transfer-v3.inc | 2 +- api-ref/source/v3/volumes-v3-snapshots.inc | 42 +- api-ref/source/v3/volumes-v3-volumes.inc | 52 +- bindep.txt | 2 + cinder/api/__init__.py | 19 +- cinder/api/api_utils.py | 91 + cinder/api/common.py | 7 +- cinder/api/contrib/admin_actions.py | 2 +- cinder/api/contrib/backups.py | 2 +- cinder/api/contrib/quota_classes.py | 4 +- cinder/api/contrib/quotas.py | 3 +- cinder/api/contrib/types_extra_specs.py | 11 +- cinder/api/contrib/types_manage.py | 6 +- cinder/api/contrib/volume_actions.py | 2 +- cinder/api/contrib/volume_image_metadata.py | 9 +- cinder/api/contrib/volume_manage.py | 2 +- cinder/api/contrib/volume_type_access.py | 3 +- cinder/api/microversions.py | 6 +- cinder/api/openstack/api_version_request.py | 14 +- cinder/api/openstack/rest_api_version_history.rst | 19 +- cinder/api/openstack/wsgi.py | 20 +- cinder/api/schemas/attachments.py | 7 +- cinder/api/v2/router.py | 93 - cinder/api/v2/volumes.py | 11 +- cinder/api/v3/attachments.py | 14 +- cinder/api/v3/default_types.py | 25 +- cinder/api/v3/group_specs.py | 10 +- cinder/api/v3/group_types.py | 6 +- cinder/api/v3/limits.py | 5 +- cinder/api/v3/router.py | 4 +- cinder/api/{v2 => v3}/snapshot_metadata.py | 0 cinder/api/v3/snapshots.py | 83 +- cinder/api/{v2 => v3}/types.py | 15 +- cinder/api/v3/views/snapshots.py | 8 +- cinder/api/{v2 => v3}/views/types.py | 17 +- cinder/api/v3/views/volumes.py | 3 + cinder/api/v3/volumes.py | 36 +- cinder/api/versions.py | 30 +- cinder/backup/api.py | 4 +- cinder/backup/chunkeddriver.py | 23 +- cinder/backup/driver.py | 10 +- cinder/backup/drivers/ceph.py | 4 +- cinder/backup/drivers/gcs.py | 14 +- cinder/backup/drivers/glusterfs.py | 5 +- cinder/backup/drivers/nfs.py | 5 +- cinder/backup/drivers/posix.py | 14 +- cinder/backup/drivers/s3.py | 14 +- cinder/backup/drivers/swift.py | 45 +- cinder/backup/manager.py | 141 +- cinder/brick/local_dev/lvm.py | 71 +- cinder/cmd/manage.py | 366 ++- cinder/common/config.py | 19 +- cinder/compute/nova.py | 8 +- cinder/context.py | 64 +- cinder/coordination.py | 39 +- cinder/db/alembic.ini | 86 + cinder/db/api.py | 42 +- cinder/db/base.py | 26 +- .../migrate_repo => legacy_migrations}/README | 0 .../migrate_repo => legacy_migrations}/__init__.py | 0 .../migrate_repo => legacy_migrations}/manage.py | 5 +- .../migrate_repo => legacy_migrations}/migrate.cfg | 0 .../legacy_migrations/versions/135_cinder_init.py | 1035 ++++++++ .../versions/136_make_vol_type_col_non_nullable.py | 10 +- .../versions/137_placeholder.py | 0 .../versions/138_placeholder.py | 0 .../versions/139_placeholder.py | 0 .../140_create_project_default_volume_type.py | 31 +- .../141_add_quota_usage_unique_constraint.py | 38 + .../versions/142_placeholder.py} | 4 +- .../versions/143_placeholder.py} | 4 +- .../versions/144_placeholder.py} | 4 +- .../versions/145_add_use_quota_fields.py | 34 + .../versions/__init__.py | 0 cinder/db/migration.py | 161 +- cinder/db/migrations/__init__.py | 0 cinder/db/migrations/env.py | 93 + cinder/db/migrations/script.py.mako | 32 + .../db/migrations/versions/921e1a36b076_initial.py | 1082 +++++++++ cinder/db/migrations/versions/__init__.py | 0 cinder/db/sqlalchemy/api.py | 444 ++-- .../migrate_repo/versions/123_cinder_init.py | 906 ------- .../migrate_repo/versions/125_placeholder.py | 22 - ..._project_resource_attribute_for_quota_usages.py | 38 - .../128_add_project_id_and_accepted_to_transfer.py | 34 - .../migrate_repo/versions/130_placeholder.py | 22 - .../migrate_repo/versions/131_placeholder.py | 22 - .../versions/132_create_default_volume_type.py | 44 - .../migrate_repo/versions/133_placeholder.py | 22 - .../migrate_repo/versions/134_placeholder.py | 22 - .../migrate_repo/versions/135_placeholder.py | 22 - cinder/db/sqlalchemy/models.py | 733 +++--- cinder/exception.py | 16 +- cinder/flow_utils.py | 7 +- cinder/group/api.py | 4 +- cinder/image/cache.py | 67 +- cinder/image/glance.py | 17 +- cinder/image/image_utils.py | 268 ++- cinder/interface/base.py | 11 +- cinder/manager.py | 37 +- cinder/message/api.py | 34 + cinder/message/message_field.py | 36 + cinder/objects/backup.py | 13 +- cinder/objects/base.py | 127 +- cinder/objects/cgsnapshot.py | 1 + cinder/objects/cleanup_request.py | 1 + cinder/objects/cluster.py | 19 +- cinder/objects/consistencygroup.py | 13 - cinder/objects/dynamic_log.py | 1 + cinder/objects/group.py | 13 +- cinder/objects/group_snapshot.py | 1 + cinder/objects/group_type.py | 1 + cinder/objects/manageableresources.py | 2 + cinder/objects/qos_specs.py | 1 + cinder/objects/request_spec.py | 16 +- cinder/objects/service.py | 21 +- cinder/objects/snapshot.py | 36 +- cinder/objects/volume.py | 53 +- cinder/objects/volume_attachment.py | 12 +- cinder/objects/volume_type.py | 17 +- cinder/opts.py | 3 +- cinder/policies/attachments.py | 53 +- cinder/policies/backups.py | 72 +- cinder/policies/base.py | 353 ++- cinder/policies/default_types.py | 45 +- cinder/policies/group_actions.py | 62 +- cinder/policies/group_snapshots.py | 52 +- cinder/policies/group_types.py | 96 +- cinder/policies/groups.py | 44 +- cinder/policies/limits.py | 11 +- cinder/policies/messages.py | 32 +- cinder/policies/quota_class.py | 29 +- cinder/policies/quotas.py | 16 +- cinder/policies/snapshot_actions.py | 18 +- cinder/policies/snapshot_metadata.py | 33 +- cinder/policies/snapshots.py | 61 +- cinder/policies/type_extra_specs.py | 64 +- cinder/policies/volume_access.py | 46 +- cinder/policies/volume_actions.py | 159 +- cinder/policies/volume_metadata.py | 107 +- cinder/policies/volume_transfer.py | 51 +- cinder/policies/volume_type.py | 209 +- cinder/policies/volumes.py | 95 +- cinder/policy.py | 4 +- cinder/quota.py | 81 +- cinder/quota_utils.py | 112 +- cinder/rpc.py | 21 +- cinder/scheduler/filter_scheduler.py | 57 +- .../scheduler/filters/instance_locality_filter.py | 20 - cinder/scheduler/manager.py | 22 +- .../v3.14/snapshot-create-response.json.tpl | 16 + .../v3.14/snapshot-show-response.json.tpl | 18 + .../v3.14/snapshot-update-response.json.tpl | 16 + .../snapshots-list-detailed-response.json.tpl | 20 + .../v3.41/snapshot-create-response.json.tpl | 17 + .../v3.41/snapshot-show-response.json.tpl | 19 + .../v3.41/snapshot-update-response.json.tpl | 17 + .../snapshots-list-detailed-response.json.tpl | 21 + .../v3.65/snapshot-create-response.json.tpl | 18 + .../v3.65/snapshot-show-response.json.tpl | 20 + .../v3.65/snapshot-update-response.json.tpl | 18 + .../snapshots-list-detailed-response.json.tpl | 22 + .../samples/versions/versions-response.json.tpl | 24 - .../encryption-type-create-request.json.tpl | 2 +- .../encryption-type-create-response.json.tpl | 2 +- .../encryption-type-show-response.json.tpl | 2 +- .../volumes/v3.13/volume-create-response.json.tpl | 35 + .../volumes/v3.13/volume-show-response.json.tpl | 39 + .../volumes/v3.13/volume-update-response.json.tpl | 37 + .../v3.13/volumes-list-detailed-response.json.tpl | 41 + .../volumes/v3.21/volume-create-response.json.tpl | 37 + .../volumes/v3.21/volume-show-response.json.tpl | 40 + .../volumes/v3.21/volume-update-response.json.tpl | 38 + .../v3.21/volumes-list-detailed-response.json.tpl | 42 + .../volumes/v3.48/volume-create-response.json.tpl | 38 + .../volumes/v3.48/volume-show-response.json.tpl | 42 + .../volumes/v3.48/volume-update-response.json.tpl | 40 + .../v3.48/volumes-list-detailed-response.json.tpl | 44 + .../volumes/v3.61/volume-create-response.json.tpl | 39 + .../volumes/v3.61/volume-show-response.json.tpl | 43 + .../volumes/v3.61/volume-update-response.json.tpl | 41 + .../v3.61/volumes-list-detailed-response.json.tpl | 45 + .../volumes/v3.63/volume-create-response.json.tpl | 40 + .../volumes/v3.63/volume-show-response.json.tpl | 44 + .../volumes/v3.63/volume-update-response.json.tpl | 42 + .../v3.63/volumes-list-detailed-response.json.tpl | 46 + .../volumes/v3.65/volume-create-response.json.tpl | 41 + .../volumes/v3.65/volume-show-response.json.tpl | 45 + .../volumes/v3.65/volume-update-response.json.tpl | 43 + .../v3.65/volumes-list-detailed-response.json.tpl | 47 + .../volumes-list-detailed-response.json.tpl | 10 +- .../unit/api/contrib/test_consistencygroups.py | 130 +- .../contrib/test_extended_snapshot_attributes.py | 4 +- .../unit/api/contrib/test_qos_specs_manage.py | 106 +- .../unit/api/contrib/test_snapshot_actions.py | 12 +- .../unit/api/contrib/test_snapshot_unmanage.py | 6 +- .../unit/api/contrib/test_types_extra_specs.py | 144 +- .../api/contrib/test_volume_encryption_metadata.py | 22 +- .../unit/api/contrib/test_volume_host_attribute.py | 14 +- .../unit/api/contrib/test_volume_image_metadata.py | 32 +- .../test_volume_migration_status_attribute.py | 14 +- .../api/contrib/test_volume_tenant_attribute.py | 16 +- .../unit/api/contrib/test_volume_type_access.py | 58 +- .../api/contrib/test_volume_type_encryption.py | 32 +- .../unit/api/{v2 => v3}/test_snapshot_metadata.py | 100 +- .../{v2/test_types.py => v3/test_types_orig.py} | 303 +-- .../unit/attachments/test_attachments_manager.py | 34 +- .../unit/policies/test_default_volume_types.py | 207 +- .../dell_emc/powerflex/test_migrate_volume.py | 16 +- .../drivers/dell_emc/powermax/powermax_data.py | 8 + .../dell_emc/powermax/test_powermax_common.py | 538 +++-- .../dell_emc/powermax/test_powermax_replication.py | 127 +- .../dell_emc/powermax/test_powermax_rest.py | 20 + .../drivers/hitachi/test_hitachi_hbsd_rest_fc.py | 199 +- .../hitachi/test_hitachi_hbsd_rest_iscsi.py | 196 +- .../unit/volume/drivers/ibm/test_storwize_svc.py | 2350 +++++++++++++++++- .../volume/drivers/inspur/instorage/test_common.py | 11 +- .../drivers/inspur/instorage/test_fc_driver.py | 5 +- .../drivers/inspur/instorage/test_iscsi_driver.py | 5 +- .../drivers/inspur/instorage/test_replication.py | 6 +- .../drivers/netapp/dataontap/client/fakes.py | 102 +- .../netapp/dataontap/client/test_client_cmode.py | 367 ++- .../unit/volume/drivers/netapp/dataontap/fakes.py | 30 + .../drivers/netapp/dataontap/test_block_base.py | 14 + .../drivers/netapp/dataontap/test_block_cmode.py | 736 +++++- .../drivers/netapp/dataontap/test_fc_cmode.py | 51 + .../drivers/netapp/dataontap/test_iscsi_cmode.py | 52 + .../drivers/netapp/dataontap/test_nfs_base.py | 41 + .../drivers/netapp/dataontap/test_nfs_cmode.py | 484 +++- .../netapp/dataontap/utils/test_data_motion.py | 297 +++ .../volume/drivers/solidfire/test_solidfire.py | 11 +- .../unit/volume/drivers/toyou/test_acs5000.py | 8 +- .../volume/drivers/vmware/test_vmware_datastore.py | 8 +- .../unit/volume/drivers/vmware/test_vmware_vmdk.py | 40 +- .../volume/drivers/vmware/test_vmware_volumeops.py | 46 +- .../unit/volume/flows/test_create_volume_flow.py | 231 +- .../unit/volume/flows/test_manage_volume_flow.py | 9 +- cinder/transfer/api.py | 4 +- cinder/utils.py | 267 ++- cinder/volume/api.py | 52 +- cinder/volume/driver.py | 61 +- cinder/volume/drivers/dell_emc/powerflex/driver.py | 8 +- cinder/volume/drivers/dell_emc/powermax/common.py | 798 ++++--- cinder/volume/drivers/dell_emc/powermax/masking.py | 2 +- .../volume/drivers/dell_emc/powermax/provision.py | 9 +- cinder/volume/drivers/dell_emc/powermax/rest.py | 19 +- cinder/volume/drivers/dell_emc/powermax/utils.py | 1 - .../volume/drivers/dell_emc/powerstore/client.py | 2 +- .../volume/drivers/dell_emc/powerstore/driver.py | 5 +- .../drivers/dell_emc/sc/storagecenter_api.py | 10 +- cinder/volume/drivers/dell_emc/unity/utils.py | 2 +- cinder/volume/drivers/dell_emc/vnx/client.py | 2 +- cinder/volume/drivers/hitachi/__init__.py | 0 cinder/volume/drivers/hitachi/hbsd_common.py | 25 +- cinder/volume/drivers/hitachi/hbsd_fc.py | 36 + cinder/volume/drivers/hitachi/hbsd_iscsi.py | 37 + cinder/volume/drivers/hitachi/hbsd_rest.py | 279 +++ cinder/volume/drivers/hitachi/hbsd_rest_api.py | 8 + cinder/volume/drivers/hitachi/hbsd_utils.py | 49 + cinder/volume/drivers/hpe/hpe_3par_base.py | 5 + cinder/volume/drivers/hpe/hpe_3par_common.py | 22 +- .../volume/drivers/ibm/storwize_svc/replication.py | 24 +- .../drivers/ibm/storwize_svc/storwize_const.py | 4 +- .../ibm/storwize_svc/storwize_svc_common.py | 792 +++++-- .../drivers/ibm/storwize_svc/storwize_svc_fc.py | 149 +- .../drivers/ibm/storwize_svc/storwize_svc_iscsi.py | 26 +- .../drivers/inspur/as13000/as13000_driver.py | 2 +- cinder/volume/drivers/linstordrv.py | 1 - cinder/volume/drivers/lvm.py | 1 - .../volume/drivers/netapp/dataontap/block_base.py | 5 + .../volume/drivers/netapp/dataontap/block_cmode.py | 344 ++- .../netapp/dataontap/client/client_cmode.py | 279 ++- cinder/volume/drivers/netapp/dataontap/fc_cmode.py | 21 +- .../volume/drivers/netapp/dataontap/iscsi_cmode.py | 6 + cinder/volume/drivers/netapp/dataontap/nfs_base.py | 23 + .../volume/drivers/netapp/dataontap/nfs_cmode.py | 235 +- .../drivers/netapp/dataontap/utils/data_motion.py | 102 + .../volume/drivers/netapp/dataontap/utils/utils.py | 1 + cinder/volume/drivers/netapp/options.py | 21 +- cinder/volume/drivers/netapp/utils.py | 8 +- cinder/volume/drivers/nfs.py | 1 - cinder/volume/drivers/nimble.py | 8 +- cinder/volume/drivers/open_e/iscsi.py | 236 +- .../drivers/open_e/jovian_common/jdss_common.py | 4 +- cinder/volume/drivers/open_e/jovian_common/rest.py | 71 +- .../drivers/open_e/jovian_common/rest_proxy.py | 4 +- cinder/volume/drivers/pure.py | 184 +- cinder/volume/drivers/quobyte.py | 2 +- cinder/volume/drivers/rbd.py | 82 +- cinder/volume/drivers/remotefs.py | 13 +- cinder/volume/drivers/san/hp/hpmsa_common.py | 8 + cinder/volume/drivers/san/hp/hpmsa_fc.py | 2 +- cinder/volume/drivers/san/hp/hpmsa_iscsi.py | 3 +- cinder/volume/drivers/spdk.py | 1 - cinder/volume/drivers/vmware/datastore.py | 23 +- cinder/volume/drivers/vmware/vmdk.py | 26 +- cinder/volume/drivers/vmware/volumeops.py | 9 +- cinder/volume/flows/api/create_volume.py | 175 +- cinder/volume/flows/common.py | 4 +- cinder/volume/flows/manager/create_volume.py | 167 +- cinder/volume/manager.py | 751 +++--- cinder/volume/rpcapi.py | 37 +- cinder/volume/targets/driver.py | 5 +- cinder/volume/volume_types.py | 3 +- cinder/volume/volume_utils.py | 336 ++- contrib/block-box/etc/api-paste.ini | 10 - .../blockstorage-user-visible-extra-specs.rst | 162 ++ .../drivers/dell-emc-powermax-driver.rst | 149 +- .../drivers/dell-emc-unity-driver.rst | 4 +- .../block-storage/drivers/dell-emc-vnx-driver.rst | 2 +- .../block-storage/drivers/hitachi-vsp-driver.rst | 2 + .../block-storage/drivers/hpe-3par-driver.rst | 68 +- .../drivers/ibm-storwize-svc-driver.rst | 19 +- .../block-storage/drivers/nimble-volume-driver.rst | 39 +- .../drivers/open-e-joviandss-driver.rst | 51 +- .../block-storage/policy-config-HOWTO.rst | 11 +- .../block-storage/policy-personas.rst | 2490 ++++++++++++++++++++ .../block-storage/samples/api-paste.ini.inc | 10 - .../block-storage/samples/policy.yaml.rst | 18 +- .../block-storage/volume-encryption.rst | 7 - .../install/cinder-controller-install-obs.rst | 80 +- .../install/cinder-controller-install-rdo.rst | 78 +- .../install/cinder-controller-install-ubuntu.rst | 80 +- driver-requirements.txt | 2 +- etc/cinder/api-paste.ini | 10 - etc/cinder/resource_filters.json | 7 +- lower-constraints.txt | 155 -- mypy-files.txt | 12 + playbooks/cinder-multibackend-matrix.yaml | 35 + rally-jobs/cinder.yaml | 90 +- ...-container-storage-policy-8d4a268ed61b9fe2.yaml | 8 + .../backup-user-messages-5ee0c7ead3def8f9.yaml | 8 + ...g-1897598-powerflex-volume-type-conversion.yaml | 9 + ...shost_calls_during_attach-528f92b44a0ff6b8.yaml | 8 + ...perswap_volume_is_failing-79a9ec2108612240.yaml | 7 + .../notes/bug-1918229-0aa9fd75c5e843d5.yaml | 8 + ...tend_to_clone_rep_volumes-015e030332f2e714.yaml | 8 + ...e-iscsi-targets-filtering-9623ac03da5c6721.yaml | 8 + ...sue-for-mirroring-volumes-31b1a9119c49e112.yaml | 7 + ...e-in-use-hyperswap-volume-95a6c033e493ee59.yaml | 7 + ...olumes_to_clone_group_fix-1cc9668ea077831e.yaml | 6 + ...m-svf-fix_addvol_gmcv_grp-caa0bc2035747d99.yaml | 6 + ...from-snapshot-skip-resize-bb5d77c5e912b5c1.yaml | 6 + ...-fix_drp_vol_create_issue-d1b75c4befb0e993.yaml | 7 + ...snapshot_clone_is_failing-c144e6b99d56de64.yaml | 7 + ...ip-properties-fetch-issue-6b443f8521cbb15c.yaml | 7 + ...g-info-for-mirror-volumes-67fbec05d803745d.yaml | 7 + ...ume_service_status_update-293dea5d0f750824.yaml | 8 + ...ties_for_empty_values_fix-a2faabbf2139195e.yaml | 6 + ...cation-support-for-V5000E-e88df9c8eb22c2a8.yaml | 8 + .../notes/bug-1939139-02ab552420813e70.yaml | 7 + ...when-plain-deletion-fails-50cef4a8a8010ba9.yaml | 24 + .../notes/bug1929429-e749f5e5a242a599.yaml | 8 + .../cinder-clone-encrypted-a28338e2b1838a63.yaml | 6 + .../cinder-manage-quota-ed4ee17f7097d11f.yaml | 6 + ...lean-file-locks-on-remove-e5898012f4114d3c.yaml | 7 + .../clean-file-locks-tool-3a62ba05ef2d2239.yaml | 10 + .../notes/drop-db_driver-opt-b644963bf3b6aced.yaml | 7 + .../notes/fix-ensure-export-3cccf107a82b35a0.yaml | 5 + ...idation-attachment-create-3488914cb52d44d2.yaml | 10 + ...ix-show-volume-non-admins-1bc5238398e73981.yaml | 7 + ...chi-generic-volume-groups-434a27b290d51bf3.yaml | 4 + ...-3par-add-alletra-9k-info-5e1d09e083d3faa9.yaml | 5 + ...nge-volumes-on-childpools-9d2217e1e6f07a0e.yaml | 5 + ...t-hyperswap-volume-extend-f578efa02314faff.yaml | 5 + ...ct-host-config-option-347e60f957458d54_new.yaml | 7 + .../notes/lvm-delete-error-f12da00c1b3859dc.yaml | 6 + .../notes/mv-use_quota-b8e010f8f68a1eaa.yaml | 14 + .../netapp-migrated-qos-c0c8aae50d010c75.yaml | 7 + ...ap-add-revert-to-snapshot-ce20810bcf094fce.yaml | 7 + .../nfs-online-snapshot-c05e6c8113bbded6.yaml | 6 + ...imble-add-alletra-6k-info-8d242a809e6044a5.yaml | 5 + ...ovisioned-capacity-option-2f8122663eec51ae.yaml | 7 + ...torage-assisted-migration-70f6fb95dbb7e580.yaml | 5 + ...iandss-enable-multiattach-b1d38ffcc53bf59c.yaml | 6 + ...owermax-create-grp-source-560139c0850e60ce.yaml | 7 + ...-delete-replication-group-76656e96262201d5.yaml | 7 + .../notes/pure-iscsi-cidrs-7195eda9f7214fce.yaml | 12 + .../notes/pure_fix_clone_cg-cfdf7d16b63882f8.yaml | 7 + .../pure_sdk_version_checks-257cb8387ed6f5f8.yaml | 11 + ..._provider_id_powervc.yaml-e794f05b0cd90f45.yaml | 6 + .../pure_storage_multiattach-f4aee3576757b2ff.yaml | 7 + ...-on-retype-with-snapshots-2d9fc7b2c75f899d.yaml | 7 + .../quota-sync-temporary-b4103ebc2c484c89.yaml | 8 + .../quota-temp-snapshots-9d032f97f80050c5.yaml | 14 + ...ota-until_refresh-updated-d35e8530f30c5522.yaml | 6 + .../quota-usage-duplicates-c00725089da7bbd8.yaml | 5 + ...hoose-correct-stripe-unit-9d317f4717533fb4.yaml | 6 + .../notes/remove-api-v2-dadd877ee5457f79.yaml | 14 + ...gurable-migration-backend-aaee5a2f808c9b36.yaml | 6 + ...retype-assisted-migration-6cdc7f9b21beb859.yaml | 7 + .../slow-get-volume-stats-91b84c6e661dc605.yaml | 7 + ...shot-in-use-without-force-86c6d74ebc9c0d60.yaml | 8 + .../support-images-api-2.11-3699b20670db1843.yaml | 12 + .../notes/switch-to-alembic-2bbe27749fde70ff.yaml | 25 + .../user-visible-extra-specs-6cf7e49c6be57a01.yaml | 16 + ...reate-delete-notification-f567cae5522852ec.yaml | 5 + .../notes/xena-known-issues-d33a6a5e10fc16f0.yaml | 54 + .../xena-policy-changes-7a563020337f6be9.yaml | 127 + .../xena-qcow2-v2-warning-9381189b1c04734f.yaml | 25 + .../xena-release-prelude-2190d8c515dbedc1.yaml | 30 + releasenotes/source/index.rst | 1 + releasenotes/source/wallaby.rst | 6 + reno.yaml | 21 +- requirements.txt | 10 +- .../defaults/main.yaml | 6 + .../defaults/main.yaml | 3 + .../save-cinder-migration-results/tasks/main.yaml | 14 + .../templates/migration_results_reporter.py.j2 | 10 + setup.cfg | 14 +- test-requirements.txt | 4 +- tools/hooks/README | 4 - tools/hooks/run_multi_backend_matrix.sh | 94 - tools/hooks/utils.sh | 10 - tools/mypywrap.sh | 4 +- tox.ini | 14 +- 625 files changed, 33408 insertions(+), 8603 deletions(-) Requirements updates -------------------- diff --git a/driver-requirements.txt b/driver-requirements.txt index 307153250..cc8da8eb5 100644 --- a/driver-requirements.txt +++ b/driver-requirements.txt @@ -12 +12 @@ krest>=1.3.0 # Apache-2.0 -purestorage>=1.6.0 # BSD +purestorage>=1.17.0 # BSD diff --git a/requirements.txt b/requirements.txt index 4e45f61b0..51e787d14 100644 --- a/requirements.txt +++ b/requirements.txt @@ -20,2 +20,2 @@ oslo.context>=3.1.1 # Apache-2.0 -oslo.db>=8.4.0 # Apache-2.0 -oslo.log>=4.4.0 # Apache-2.0 +oslo.db>=11.0.0 # Apache-2.0 +oslo.log>=4.5.0 # Apache-2.0 @@ -24 +24 @@ oslo.middleware>=4.1.1 # Apache-2.0 -oslo.policy>=3.6.2 # Apache-2.0 +oslo.policy>=3.8.1 # Apache-2.0 @@ -51 +51 @@ six>=1.15.0 # MIT -SQLAlchemy>=1.3.19 # MIT +SQLAlchemy>=1.4.23 # MIT @@ -59 +59 @@ oslo.vmware>=3.8.0 # Apache-2.0 -os-brick>=4.3.0 # Apache-2.0 +os-brick>=5.0.1 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 591debf04..ac0964da9 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -17 +17 @@ psycopg2>=2.8.5 # LGPL/ZPL -SQLAlchemy-Utils>=0.36.8 # BSD License +SQLAlchemy-Utils>=0.37.8 # BSD License @@ -24 +24 @@ doc8>=0.8.1 # Apache-2.0 -mypy>=0.782 # MIT +mypy>=0.910 # MIT