We are pumped to announce the release of: swift 2.36.0 This release is part of the flamingo release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.36.0 ^^^^^^ New Features ************ * S3 API * Added support for aws-chunked transfers. Recent AWS clients recently began defaulting to this mode. See also: Transferring Payload in Multiple Chunks (Chunked Upload). * Added support for verifying additional checksums during upload. All algorithms currently supported by AWS are supported: CRC64NVME, CRC32, CRC32C, SHA1, and SHA256. See also: Checking object integrity in Amazon S3. Note that some algorithms require the availability of additional libraries: ISA-L or anycrc. * Added support for create-without-overwrite conditional writes. * Let clients request heartbeats during COPYs by including the query parameter "heartbeat=on". With heartbeating turned on, the proxy will start its response immediately with 202 Accepted then send a single whitespace character periodically until the request completes. At that point, a final summary chunk will be sent which includes a "Response Status" key indicating success or failure. * Labeled metrics * Added support for emitting labeled statsd metrics in a variety of formats. Middleware authors should see the LabeledStatsdClient documentation for more information. * Instrumented proxy-logging with labeled metrics. See "proxy- server .conf-sample" for more information. * Instrumented the object-server with labeled metrics. See "object- server.conf-sample" for more information. * Added "access_user_id" logging field; out-of-tree auth middlewares should use "environ['swift.access_logging']['user_id']" to populate this field. * Introduced an extensible ring format. This allows both more than 65,536 devices and more data structures to be in a ring. For more information, see Ring File Formats -- Ring v2. * Python 3.13 (with the GIL enabled) is now supported. Free-threaded builds remain untested. Upgrade Notes ************* * Removed support for Python 3.6. * Removed support for pickled ring files. These have not been written since Swift 1.7.0. Deprecation Notes ***************** * Flamingo (2025.2) will be the final stable release to support Python 3.7 and 3.8. Bug Fixes ********* * S3 API * Fixed HTTP framing issues when returning errors for a request with "Expect: 100-continue". * Improved various error messages to better imitate AWS responses. * SSYNC connections are now promptly terminated when subrequests timeout. * Fixed a recursion error in the account-quota middleware. * Fixed an error in "invalidate_hash" when the partition is deleted while waiting for the partition lock. * Various other minor bug fixes and improvements. Changes in swift 2.35.0..2.36.0 ------------------------------- 82cb5a5d7 AUTHORS/CHANGELOG for 2.36.0 e13f4abcd tests: Skip some tests if crc32c is not available 21325988d CI: Remove a bunch of unnecessary bindep profiles 3db8e2d05 Clean up some py36 infra 1ed7b71bb Update py3-constraints.txt a18fb08b4 Switch py39 jobs to use py3-constraints.txt 32eaab20b proxy-logging: create field for access_user_id aa3a4eeb2 docs: Fix python3 package names c4cc83c5e s3api compat tests: stop asserting DisplayName in Owner 81df05a9c checksums: Ignore broken pyeclib installs 4d2426591 CI: Stop using bionic de81a844f Revert "Remove unused MANIFEST.in" 7f3e76129 zuul: run py3.13 unit tests in the gate 5b2e4e00c ssync unit test: fix intermittent failure 74274ec8b checksum.py: fail gracefully if pyeclib is broken 93b88540d Timestamp: fix inconsistent raw and normal values 683218c52 Ring v2 follow-up a770881b5 Fix another way mw may encouter invalid swift paths 1dc3307ea ssync-receiver: terminate session if subreq read times out 86a1acc9e crypto: Fix traceback on non-utf8, non-swift paths 06a632979 Fix recursion error in account_quota middleware fd342b919 sharder: make gap and overlap warning logs shorter ae062f8b0 ring: Introduce a v2 ring format 0417979ca Drop support for old pickled rings 404e1f273 s3api: Add support for crc64nvme checksum calculation 61c0bfcf9 s3api: add more assertions w.r.t. S3 checksum BadDigest 351ee7279 s3api: add compat test sending too much body with checksum be56c1e25 s3api: Validate additional checksums on upload de579d63d Imported Translations from Zanata 1a27d1b83 s3api: fix multi-upload BadDigest error a555a8593 diskfile: Stop including traceback on quarantine 53b66155a test: use a tempdir in TestRingData 3dba68100 Fix traceback in invalidate_hash 89ad202f7 Update SAIO document e9da48daa Fix swift-dsvm-functional for devstack 9754eff02 Use built-in implementation to get utc timezone c177e7dcf Clarify why there's a ShardRange.__hash__ 005d69d1a Drop remaining skip check for Python < 3 2e14051cb tests: Fix some connection-closed testing on OS X 28db9bbcd tests: Fix test_LoggerFileObject_recursion 49ae2aa94 CI: Skip grenade-skip-level-always similar to tempest 962084ded s3 compat tests: sanitize object listings 313959ae9 Object-server: change labeled timing metrics sample rate for debugging requests 38ad3a486 CI: run S3 cross-compat tests as part of DSVM jobs 3ff6b34a3 tests: Keep port number in valid range aa5bc0198 tests: Speed up statsd test_methods_are_no_ops_when_not_enabled 877c936e2 tests: Reduce test time on OSX c66c0bfd2 Object-server: add labeled timing metrics for other object server requests 33b17742a s3api: more test cases for conditional writes. 82b196447 Add labeled metrics to proxy-logging b735b3d03 object-server: return 503 not 404 if meta or data file unlinked edd5eb29d s3api: Allow PUT with `if-none-match: *` ec6e8bd20 common: add http exception handling for base storage server timing stats 296e894f7 Remove installation guide for openSUSE/SLES 43aa7ec26 add pyproject.toml to support pip 23.1 a2725ea99 CI: Move lower-constraints job to py37 33f409ceb CI: run func tests during rolling upgrade job 020b5a78a trivial: statsd tests: remove references to delegate methods 7a3b7373b Add config option for whether to skip s3_acl-requiring tests 0714557aa Drop support for py36 b65855d71 utils: Add CRCHasher and crc32c implementation f9ac22971 Add support of Sigv4-streaming 33b89492e Remove unused MANIFEST.in 9a445503e Implement heartbeat response for COPY request e708943a1 CI: update rolling-upgrade jobs ab6e05922 Object-server: add labeled timing metrics for object REPLICATE request b07843e92 Fix the inconsistent use of printing errors f8efa83da Remove tags from README 74030236a tempauth: Support fernet tokens b80de7cc3 Test under py313 51c8ea018 s3api: Remove some useless decoding 1ca073ce1 Let WSGI know the length ab5c742e2 s3api: make MPU part error response message same as S3 a93e420d3 s3request: refactor to introduce SigChecker classes addcc1e32 s3request: refactor error handling while reading input dd0ba770f Add skip reason next to test name in zuul output 63224cfd0 tests: rename zuul jobs that test s3api 37a25b340 tests: Remove BaseFakeStatsdClient.get_increment_counts 563c1ca65 DRY out AbstractStatsdClient public methods 405440a2e tests: Display extra test summary info ed0ef3571 tests: add LabeledStatsdClient sample_rate kwarg coverage 11257eda5 proxy_logging: remove unnecessary condition 769e482ea tests: provide FakeLabeledStatsdClient for debug 0e2791a88 Remove deprecated statsd label_mode 7e5235894 stats: API for native labeled metrics 05143a99f Update master for stable/2025.1 688656a09 Clarify that get_ppid is Linux-only 2f74376e7 Make FakeSwiftCall.DUMMY_VALUE a bespoke singleton ff4551944 tests: Use subTest Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 + .mailmap | 1 + .zuul.yaml | 102 +- AUTHORS | 4 + CHANGELOG | 70 + README.rst | 5 - etc/magic | 20 + etc/proxy-server.conf-sample | 60 +- lower-constraints.txt | 3 +- py3-constraints.txt | 231 +- pyproject.toml | 3 + .../notes/release-2.36.0-9f4b539db40bffd7.yaml | 88 + releasenotes/source/2025.1.rst | 6 + releasenotes/source/index.rst | 1 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 140 +- requirements.txt | 2 +- roles/additional-tempauth-users/tasks/main.yaml | 8 +- roles/dsvm-additional-middlewares/tasks/main.yaml | 18 +- setup.cfg | 3 +- swift/account/server.py | 4 +- swift/cli/info.py | 12 +- swift/cli/manage_shard_ranges.py | 2 +- swift/cli/ringbuilder.py | 91 +- swift/common/base_storage_server.py | 75 +- swift/common/exceptions.py | 12 + swift/common/middleware/account_quotas.py | 74 +- swift/common/middleware/bulk.py | 54 +- swift/common/middleware/catch_errors.py | 56 +- swift/common/middleware/copy.py | 102 +- swift/common/middleware/crypto/decrypter.py | 12 +- swift/common/middleware/crypto/keymaster.py | 17 +- swift/common/middleware/crypto/kmip_keymaster.py | 4 +- swift/common/middleware/crypto/kms_keymaster.py | 5 +- swift/common/middleware/keystoneauth.py | 4 + swift/common/middleware/proxy_logging.py | 119 +- swift/common/middleware/ratelimit.py | 2 +- .../middleware/s3api/controllers/multi_upload.py | 37 +- swift/common/middleware/s3api/exception.py | 78 + swift/common/middleware/s3api/s3api.py | 3 +- swift/common/middleware/s3api/s3request.py | 1088 ++++++-- swift/common/middleware/s3api/s3response.py | 32 +- .../s3api/schema/complete_multipart_upload.rng | 25 + swift/common/middleware/s3api/utils.py | 5 +- swift/common/middleware/slo.py | 11 +- swift/common/middleware/tempauth.py | 167 +- swift/common/request_helpers.py | 46 + swift/common/ring/builder.py | 87 +- swift/common/ring/composite_builder.py | 9 +- swift/common/ring/io.py | 650 +++++ swift/common/ring/ring.py | 300 +- swift/common/ring/utils.py | 68 + swift/common/statsd_client.py | 551 +++- swift/common/swob.py | 5 +- swift/common/utils/__init__.py | 92 +- swift/common/utils/checksum.py | 266 ++ swift/common/utils/logs.py | 6 +- swift/common/utils/timestamp.py | 30 +- swift/container/server.py | 4 +- swift/container/sharder.py | 13 +- swift/obj/diskfile.py | 94 +- swift/obj/reconstructor.py | 1 + swift/obj/server.py | 85 +- swift/obj/ssync_receiver.py | 153 +- test-requirements.txt | 1 + test/__init__.py | 24 - test/debug_logger.py | 81 +- test/functional/__init__.py | 44 +- test/functional/s3api/test_multi_upload.py | 60 + test/functional/s3api/test_object.py | 54 +- test/probe/test_reconstructor_rebuild.py | 33 +- test/probe/test_sharder.py | 81 +- test/s3api/__init__.py | 17 +- test/s3api/test_conditional_writes.py | 142 + test/s3api/test_input_errors.py | 2884 +++++++++++++++++--- test/s3api/test_mpu.py | 10 +- test/s3api/test_object_checksums.py | 600 ++++ test/s3api/test_service.py | 75 +- test/s3api/test_versioning.py | 62 +- test/sample.conf | 11 +- test/unit/__init__.py | 39 +- test/unit/account/test_auditor.py | 2 +- test/unit/account/test_reaper.py | 6 +- test/unit/cli/test_default_output.stub | 2 +- test/unit/cli/test_default_output_id_assigned.stub | 2 +- test/unit/cli/test_default_sorted_output.stub | 10 +- test/unit/cli/test_info.py | 8 +- test/unit/cli/test_ipv6_output.stub | 2 +- test/unit/cli/test_ringbuilder.py | 173 +- .../common/middleware/crypto/test_decrypter.py | 26 +- .../common/middleware/crypto/test_keymaster.py | 2 +- test/unit/common/middleware/helpers.py | 12 +- test/unit/common/middleware/s3api/__init__.py | 5 +- test/unit/common/middleware/s3api/test_bucket.py | 10 +- .../common/middleware/s3api/test_multi_upload.py | 182 +- test/unit/common/middleware/s3api/test_obj.py | 23 +- test/unit/common/middleware/s3api/test_s3api.py | 201 +- .../unit/common/middleware/s3api/test_s3request.py | 1416 +++++++++- .../common/middleware/s3api/test_s3response.py | 24 +- test/unit/common/middleware/s3api/test_utils.py | 11 +- test/unit/common/middleware/test_account_quotas.py | 16 + .../common/middleware/test_backend_ratelimit.py | 4 +- test/unit/common/middleware/test_bulk.py | 4 +- test/unit/common/middleware/test_catch_errors.py | 26 +- test/unit/common/middleware/test_copy.py | 141 +- test/unit/common/middleware/test_formpost.py | 2 +- test/unit/common/middleware/test_list_endpoints.py | 8 +- test/unit/common/middleware/test_proxy_logging.py | 790 ++++-- test/unit/common/middleware/test_tempauth.py | 150 +- test/unit/common/middleware/test_tempurl.py | 2 +- test/unit/common/ring/test_builder.py | 80 +- test/unit/common/ring/test_composite_builder.py | 7 +- test/unit/common/ring/test_io.py | 238 ++ test/unit/common/ring/test_ring.py | 627 ++++- test/unit/common/ring/test_utils.py | 4 - test/unit/common/test_base_storage_server.py | 235 +- test/unit/common/test_statsd_client.py | 631 ++++- test/unit/common/test_utils.py | 106 +- test/unit/common/utils/test_checksum.py | 417 +++ test/unit/common/utils/test_config.py | 3 +- test/unit/common/utils/test_logs.py | 2 +- test/unit/common/utils/test_timestamp.py | 30 +- test/unit/container/test_backend.py | 7 +- test/unit/container/test_reconciler.py | 5 +- test/unit/container/test_replicator.py | 35 +- test/unit/container/test_server.py | 10 +- test/unit/container/test_sharder.py | 150 +- test/unit/container/test_updater.py | 29 +- test/unit/helpers.py | 33 +- test/unit/obj/test_diskfile.py | 128 +- test/unit/obj/test_expirer.py | 38 +- test/unit/obj/test_reconstructor.py | 71 +- test/unit/obj/test_replicator.py | 42 +- test/unit/obj/test_server.py | 568 +++- test/unit/obj/test_ssync.py | 87 +- test/unit/obj/test_ssync_receiver.py | 398 ++- test/unit/obj/test_ssync_sender.py | 50 +- test/unit/obj/test_updater.py | 120 +- test/unit/proxy/controllers/test_base.py | 8 +- test/unit/proxy/controllers/test_container.py | 17 +- test/unit/proxy/test_server.py | 52 +- tools/playbooks/multinode_setup/run.yaml | 1 - tox.ini | 25 +- 160 files changed, 14707 insertions(+), 3044 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 6105b8e4a..4ae431c95 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +6 @@ eventlet>=0.25.0,!=0.34.3 # MIT -greenlet>=0.3.3 +greenlet>=0.4.14 diff --git a/test-requirements.txt b/test-requirements.txt index 9b8ec065b..28697926a 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8,0 +9 @@ pytest>=4.6.11 # MIT +pytest-subtests>=0.3.0 # MIT