We are psyched to announce the release of: swift 2.31.0: OpenStack Object Storage This release is part of the antelope release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.31.0 ^^^^^^ New Features ************ * Added support for Python 3.10. * Added an optional "backend_ratelimit" middleware for backend servers. See the backend server sample configuration files for more information. * Sharding improvements * Added a "merge" subcommand to "swift-manage-shard-ranges" to merge arbitrary shard ranges into a container DB. Minimal safety checks are performed; it should only be used for emergency shard range manipulation by expert users. * Warnings are now emitted when sharding appears to have become stuck. Use the new "container_sharding_timeout" option to configure the "stuck" threshold; the default is 48 hours. * Metrics improvements * Added timing stats for memcached operations. * Renamed and improved the granularity of shard range cache and backend stats. Metrics dashboards may need to be updated. * Emit stats when backend nodes are error-limited. * Added the ability to configure a chance to skip checking memcache when querying account and container information. This allows some fraction of traffic to go to disk and refresh memcache before the key ages out. Recommended values for the new "account_existence_skip_cache_pct" and "container_existence_skip_cache_pct" options are in the range of 0.0 to 0.01. * Absolute-form request targets are now accepted. This enables access for certain clients and SDKs (including some older versions of rclone that were using an old version of aws-sdk-go). Upgrade Notes ************* * Static large object segments may now be deleted asynchronously by default. Operators may return to the old behavior by disabling the "allow_async_delete" option in the "[filter:slo]" section in their proxy-server.conf. Security Issues *************** * Fixed a security issue in how "s3api" handles XML parsing that allowed authenticated S3 clients to read arbitrary files from proxy servers. Refer to CVE-2022-47950 for more information. Bug Fixes ********* * S3 API improvements * Fixed a server error when handling malformed CompleteMultipartUpload requests. * Improved error reporting when attempting to set invalid "X-Delete- At" or "X-Delete-After" values via the S3 API. * Sharding improvements * Sync more shard ranges from the root database to the shards. This helps ensure shard range repairs effected at the root make their way to shards that would otherwise be stuck trying to further divide into sub-shards. * Improved performance of "delimiter" listings for sharded containers. * Added more safety checks to the "repair" subcommand of "swift- manage-shard-ranges". * Better handle "EOFError" and "KeyboardInterrupt" when prompting for input in "swift-manage-shard-ranges". * Stop warning about transient overlaps when auditing shard ranges. * Fixed a path-rewriting bug introduced in Python 3.7.14, 3.8.14, 3.9.14, and 3.10.6 that could cause some "domain_remap" requests to be routed to the wrong object. * Fixed a server error when attempting to access data in a deleted container that had an erasure-coded storage policy. * Improved error messages to clients that encounter errors using the "formpost" middleware. * Removed some inappropriate error-suppression when locking account and container databases. * Improved server start-up time when using multiple workers. * Removed some unnecessary locking when logging. * Added some basic object-metadata validation; invalid diskfiles will be quarantined via the auditor or reconstructor. * Enhanced logging when error-limiting a backend node. * Various other minor bug fixes and improvements. Changes in swift 2.30.0..2.31.0 ------------------------------- 90f9a479b Authors/ChangeLog for 2.31.0 5de745c2b Add Kota's private email address to mailmap 941b9082d Proxy: move '_get_update_shard' from base class to child class. 2f32d0770 swift_proxy: add memcache skip success/error stats for shard range. 30ffafb3e Clean up some cruft 3550e00dd tests: Ensure XXE injection tests have config loaded 51a944850 Skip coverage reports when running pytest directly 908595ee0 Don't run reno as part of building an sdist cc1c649de Clean up project URLs for PyPI dd9b687f7 Fix docs build b8467e190 s3api: Prevent XXE injections 69b18e3c5 tests: Remove references to soft_lock 930744f36 Declare py310 support fb477d17a CI: Remove nodeset pin for DSVM jobs ee12a11e7 Add attrs to lower-constraints b4124e0cd Memcached: add timing stats to set/get and other public functions f77172e2a CI: pin tox at the project level ec9504733 Sharder: add a new probe test for the case of slow parent sharding. f6ac7d449 Tolerate absolute-form request targets 2d7c1dc6d CI: Fix our usage of tox 20b48a690 Clean up a bunch of deprecation warnings 2d0fea6a5 CI: Add py310 unit test job ef155bd74 Switch to pytest 884f5538f Inline parse_request from cpython 597887ded Extract SwiftHttpProtocol to its own module 4cba97d7b Malformed CompleteMultipartUpload request should 400 d363236a2 s3api errors for unsupported headers x-delete-at, x-delete-after 001d931e6 sharder: update own_sr stats explicitly 07b909c26 s-m-s-r: handle EOFError and KeyboardInterrupt 861d13c51 DB locks shouldn't squelch errors 662ef2808 trivial: fix flakey account/test_auditor.py assertion a1939cba0 proxy: add test for ContainerController._GET_using_cache bd5d09903 proxy: refactor ContainerController._GET_using_cache 7a0118fca proxy-server: include suppression time in error limit log 230b467af sharding: avoid transient overlaps while auditing. 5a9f5f521 Temporary pin the swift-dsvm-functional nodeset to Focal 983622d51 account_auditor: fix warning string 26c86c0e4 db_auditor: remove logging translation 2bcf3d1a8 sharder: merge shard shard_ranges from root while sharding 12df65e37 Imported Translations from Zanata 38124221d Proxy: add metrics related to error limiter. d1b2bbdcf Mark rolling-upgrade job non-voting 623024848 proxy: extract response error handling to single method 162847d15 tests: Tolerate NoSuchBucket errors when cleaning up f08b8e0af wsgi: Start workers in parallel, rather than serially 052abed75 Move base CI job to jammy c4ad66b49 Sharder: add more logs during container cleaving. 3571cdf2f CI: Install pip from pinned upstream for py2 probe tests a8516e151 Refactor memcache config and MemcacheRing loading 51730f127 proxy: refactor error limiter to a class a7af17394 Clean up some replication error messages 326e68d70 Fix a formpost reponse bug 79e8f56ac Imported Translations from Zanata cc033154a py2constraints: pin PasteDeploy version 4ed2b89cb Sharder: warn when sharding appears to have stalled. e09d4bcf4 Stop locking in our SysLogHandler 07b5f139a playbooks: replace ansible_ssh_user with ansible_user a46f2324a sharder: always merge child shard ranges fetched from root e838d8a94 swift-drive-audit: reload systemd after editing fstab f15b92084 sharder: always get ranges from root while shrinking 429702e30 sharder: refactor _audit_shard_container 7b345e754 Update master for stable/zed a425f06ee Imported Translations from Zanata 6bcb50985 sharder: trivial tidy up of unit tests 94fb7a0e7 Set irrelevant-files for openstacksdk-functional-devstack job 0860db1f6 Fix test.probe.brain CLI 9bdb77a38 Fixes Storage-Policy AttributeError in proxy a53270a15 swift-manage-shard-ranges repair: check for parent-child overlaps. 032d7aa05 ShardName: add test and clarify docstrings 73bce6331 ShardRange: add is_child_of method 745a7f04f Add validation method for metadata in ECDiskfile 3e6650f99 Fix misuse of assertTrue d2edf6646 swift-manage-shard-ranges: add 'merge' subcommand 5c6407bf5 proxy: Add a chance to skip memcache for get_*_info calls c4ff49c32 CI: Add rolling upgrade job coming from stable/yoga a9177a4b9 Add note about rsync_bwlimit suffixes da52e5675 Migrate CentOS Stream 8 FIPS job to CentOS Stream 9 d0feee190 Add MPU to s3api tests 17664cfb9 Use daemon property instead of setDaemon method 61624ab83 swift-manage-shard-ranges repair: ignore recent overlaps 38866a396 Optimize ShardRanges a little 85d063b20 ring: Have RingBuilder.debug return to old state 25463b253 sharding: Skip shards that can't include any new subdir entries ccaf49a00 Add backend rate limiting middleware fa1058b6e slo: Default allow_async_delete to true Diffstat (except docs and test files) ------------------------------------- .mailmap | 4 +- .zuul.yaml | 155 +- AUTHORS | 6 +- CHANGELOG | 95 ++ CONTRIBUTING.rst | 4 +- bin/swift-drive-audit | 4 + bindep.txt | 2 +- etc/account-server.conf-sample | 14 +- etc/container-server.conf-sample | 20 +- etc/object-server.conf-sample | 17 +- etc/proxy-server.conf-sample | 4 +- lower-constraints.txt | 8 +- py2-constraints.txt | 7 +- py36-constraints.txt | 3 - .../notes/2_31_0_release-77e6b20dfba3b32c.yaml | 118 ++ releasenotes/source/index.rst | 2 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 164 ++- releasenotes/source/zed.rst | 6 + setup.cfg | 23 +- swift/__init__.py | 10 +- swift/account/auditor.py | 7 +- swift/account/reaper.py | 17 +- swift/cli/dispersion_report.py | 7 +- swift/cli/manage_shard_ranges.py | 222 ++- swift/common/db.py | 19 +- swift/common/db_auditor.py | 19 +- swift/common/error_limiter.py | 93 ++ swift/common/http_protocol.py | 326 +++++ swift/common/memcached.py | 118 +- swift/common/middleware/backend_ratelimit.py | 86 ++ swift/common/middleware/formpost.py | 26 +- swift/common/middleware/memcache.py | 96 +- .../middleware/s3api/controllers/multi_upload.py | 20 +- swift/common/middleware/s3api/etree.py | 2 +- swift/common/middleware/s3api/exception.py | 4 - swift/common/middleware/s3api/s3api.py | 3 + swift/common/middleware/s3api/s3request.py | 18 +- swift/common/middleware/slo.py | 4 +- .../common/middleware/versioned_writes/__init__.py | 3 + swift/common/ring/builder.py | 4 +- swift/common/swob.py | 3 + swift/common/utils.py | 354 ++++- swift/common/wsgi.py | 266 +--- swift/container/backend.py | 108 +- swift/container/sharder.py | 510 +++++-- swift/container/updater.py | 12 +- swift/obj/auditor.py | 4 + swift/obj/diskfile.py | 14 + swift/obj/reconstructor.py | 19 +- swift/obj/replicator.py | 19 +- swift/obj/ssync_sender.py | 18 +- swift/obj/updater.py | 19 +- swift/proxy/controllers/base.py | 162 +-- swift/proxy/controllers/container.py | 297 ++-- swift/proxy/controllers/obj.py | 113 +- swift/proxy/server.py | 124 +- test-requirements.txt | 7 +- test/__init__.py | 55 +- test/functional/__init__.py | 3 +- test/functional/s3api/test_object.py | 51 +- test/functional/s3api/test_presigned.py | 30 + test/functional/s3api/test_xxe_injection.py | 231 +++ test/probe/brain.py | 5 +- test/probe/common.py | 22 + test/probe/test_sharder.py | 335 ++++- test/s3api/__init__.py | 4 + test/s3api/test_mpu.py | 140 ++ test/s3api/test_request_target_style.py | 56 + test/unit/__init__.py | 4 - test/unit/account/test_auditor.py | 6 +- test/unit/cli/test_manage_shard_ranges.py | 625 +++++++- test/unit/common/middleware/helpers.py | 4 +- test/unit/common/middleware/s3api/__init__.py | 5 + .../common/middleware/s3api/test_multi_delete.py | 40 + test/unit/common/middleware/s3api/test_s3token.py | 12 +- .../common/middleware/test_backend_ratelimit.py | 170 +++ test/unit/common/middleware/test_formpost.py | 32 +- test/unit/common/middleware/test_memcache.py | 214 +-- test/unit/common/middleware/test_slo.py | 10 +- test/unit/common/test_db.py | 23 +- test/unit/common/test_db_replicator.py | 93 +- test/unit/common/test_error_limiter.py | 99 ++ test/unit/common/test_http_protocol.py | 454 ++++++ test/unit/common/test_manager.py | 6 +- test/unit/common/test_memcached.py | 276 ++++ test/unit/common/test_storage_policy.py | 5 +- test/unit/common/test_utils.py | 537 ++++++- test/unit/common/test_wsgi.py | 335 +---- test/unit/container/test_backend.py | 125 +- test/unit/container/test_server.py | 15 +- test/unit/container/test_sharder.py | 1526 +++++++++++++++++--- test/unit/helpers.py | 8 +- test/unit/obj/common.py | 1 + test/unit/obj/test_auditor.py | 31 +- test/unit/obj/test_diskfile.py | 36 +- test/unit/obj/test_replicator.py | 24 +- test/unit/obj/test_server.py | 10 +- test/unit/obj/test_ssync.py | 1 + test/unit/proxy/controllers/test_base.py | 56 +- test/unit/proxy/controllers/test_container.py | 583 +++++++- test/unit/proxy/controllers/test_obj.py | 26 +- test/unit/proxy/test_server.py | 619 ++++++-- tools/playbooks/common/install_dependencies.yaml | 24 +- tools/playbooks/multinode_setup/common_config.yaml | 4 +- tools/playbooks/multinode_setup/make_rings.yaml | 8 +- tools/playbooks/multinode_setup/run.yaml | 2 +- .../templates/make_multinode_rings.j2 | 2 +- .../saio_single_node_setup/setup_saio.yaml | 14 +- tools/test-setup.sh | 12 + tox.ini | 43 +- 117 files changed, 8657 insertions(+), 2648 deletions(-) Requirements updates -------------------- diff --git a/test-requirements.txt b/test-requirements.txt index 7eb48af72..488080ae5 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -7,4 +7,3 @@ hacking>=2.0,<2.1.0 # Apache-2.0 -coverage>=3.6 # Apache-2.0 -nose>=1.3.7 # LGPL -nosexcover>=1.0.10 # BSD -nosehtmloutput>=0.0.3 # Apache-2.0 +coverage>=5.0.4 # Apache-2.0 +pytest>=4.6.11 # MIT +pytest-cov>=2.12.1 # MIT