We are happy to announce the release of: swift 2.21.0: OpenStack Object Storage This release is part of the stein release series. Download the package from: https://tarballs.openstack.org/swift/ For more details, please see below. 2.21.0 ^^^^^^ New Features ************ * Change the behavior of the EC reconstructor to perform a fragment rebuild to a handoff node when a primary peer responds with 507 to the REPLICATE request. This changes EC to match the existing behavior of replication when drives fail. After a rebalance of EC rings (potentially removing unmounted/failed devices), it's most IO efficient to run in handoffs_only mode to avoid unnecessary rebuilds. * O_TMPFILE support is now detected by attempting to use it instead of looking at the kernel version. This allows older kernels with backported patches to take advantage of the O_TMPFILE functionality. * Add slo_manifest_hook callback to allow other middlewares to impose additional constraints on or make edits to SLO manifests before being written. For example, a middleware could enforce minimum segment size or insert data segments. * Fixed an issue with multi-region EC policies that caused the EC reconstructor to constantly attempt cross-region rebuild traffic. * Fixed an issue where S3 API v4 signatures would not be validated against the body of the request, allowing a replay attack if request headers were captured by a malicious third party. * Display crypto data/metadata details in swift-object-info. * formpost can now accept a content-encoding parameter. * Fixed an issue where multipart uploads with the S3 API would sometimes report an error despite all segments being upload successfully. * Multipart object segments are now actually deleted when the multipart object is deleted via the S3 API. * Swift now returns a 503 (instead of a 500) when an account auto- create fails. * Fixed a bug where encryption would store the incorrect key metadata if the object name starts with a slash. * Fixed an issue where an object server failure during a client download could leave an open socket between the proxy and client. * Fixed an issue where deleted EC objects didn't have their on-disk directories cleaned up. This would cause extra resource usage on the object servers. * Fixed issue where bulk requests using xml and expect 100-continue would return a malformed HTTP response. * Various other minor bug fixes and improvements. Bug Fixes ********* * Fixed a cache invalidation issue related to GET and PUT requests to containers that would occasionally cause object PUTs to a container to 404 after the container had been successfully created. * Removed a race condition where a POST to an SLO could modify the X -Static-Large-Object metadata. * Fixed rare socket leak on range requests to erasure-coded objects. * Fix SLO delete for accounts with non-ASCII names. * Fixed an issue in COPY where concurrent requests may have copied the wrong data. * Fixed time skew when using X-Delete-After. * Send ETag header in 206 Partial Content responses to SLO reads. * Fix SLO delete for accounts with non-ASCII names. * Fixed an issue in COPY where concurrent requests may have copied the wrong data. * Fixed a bug in how Swift uses eventlet that was exposed under high concurrency. * Prevent PyKMIP's kmip_protocol logger from logging at DEBUG. Previously, some versions of PyKMIP would include all wire data when the root logger was configured to log at DEBUG; this could expose key material in logs. Only the kmip_keymaster was affected. * Fixed an issue where a failed drive could prevent the container sharder from making progress. * Fixed a bug in how Swift uses eventlet that was exposed under high concurrency. Changes in swift 2.20.0..2.21.0 ------------------------------- 179fa7ccd authors/changelog update for 2.21.0 release 64eec5fc9 Fix how we UTF-8-ify func tests fe3a20f2e Remove uncalled function adc568c97 Fix bulk responses when using xml and Expect 100-continue 585bf40cc Simplify empty suffix handling e5eb673cc Stop monkey-patching mimetools d6af42b6b Clean up how we walk through ranges in ECAppIter c9773bfd2 Add non-voting py37 unit test job 95da1d97b Fix py35 unit test job 53b56b655 crediting contributors to the un-landed hummingbird branch fa678949a Fix quoting for large objects a30a47775 Stop overwriting reserved term 74664af7e Fix a typo 13e7f3641 Do not use self in classmethod e1a12dc3d Refactor write_affinity DELETE handling d74885176 s3token: Add note about config change when upgrading from swift3 d185b607b docs: clean up SAIO formatting 860ae1e00 py3: port proxy account controller 8b519d1ab Clean up func tests ahead of py3 45994e4d3 py3: port object updater 4ac81ebbd py3: fix copying unicode names 5d4303edb manage-shard-ranges: nicer message if we can't get_info() 349c54998 py3: port object auditor 61e6ac0eb py3: port formpost middleware 04a0ff2e9 added changelog entries from stable releases 5409c4f34 Make test_multiphase_put_drains_extra_commit_junk_disconnect less flakey be3e21582 py3: display help if no subcommand provided to manage-shard-ranges baf18edc0 Clean up account-reaper a bit b4e470076 py3: port some more middleware tests 3be384c81 py3: port proxy container controller 575538b55 py3: port the container ce261b12e py3: port versioned_writes f4689dd22 probe tests: wait to start replicators until after verifying initial state db9c90584 py3: port copy middleware 1e3f8a0e5 Address some review comments 926a02413 Fix up flakey TestContainer.test_PUT_bad_metadata 78fb722ab Address some review comments 693f931a5 Fixing target for cross-ref 'get_data_dir' 002d21991 Make get_data/async/tmp_dir explicit 9b3ca9423 Fix decryption for broken objects ac01d186b Leave less garbage in /var/tmp bb1a2d456 Display crypto data/metadata details in swift-object-info ea8e545a2 Rebuild frags for unmounted disks 8a6159f67 Stop using duplicate dev IDs in write_fake_ring 828a241ca py3: adapt proxy base controller 43103319d encryption: Stop being cutesy with os.path.join() c6d213f82 Consolidate retry code in functest client 988e71923 py3: port object controller in proxy 90b161b74 Add content-encoding parameter to formpost middleware 3e2b8c95f Return 503 when account auto-create fails f319c412a Fix flakey symlink setup/teardown fb0e7837a Cleanup EC and SSYNC frag index parameters 2bd7b7a10 py3 object-server follow-ups dd379e7f5 func tests: Be willing to retry PUTs d3946d970 Fix flakey func test teardown 32d2af06e Fix flakey func test setup 61c9aa4bf probetests: make negative assertion more meaningful 7ed12fa6c Move KMIP conf validation out of _get_root_secret c0dbf5b88 sharding: Make replicator logging less scary b2b1f96b0 docs: clean up Object Versioning page 1f173fced s3api: fix up flakey LastModified assertions 5037ab577 Python3: fix test/unit/obj/test_replicator.py 9b6080536 py3: proxy some logger properties in LogAdapter 0e81ffd1e Fix socket leak on object-server death 066873183 Change how O_TMPFILE support is detected ca98b5b37 Quiet down a unittest c4363aa6e Python3: Fix test/unit/common/test_container_sync_realms.py 700fcc735 Remove duplicate statement 1d4309dd7 misc test cleanup ca4d6ec99 Save original index in the determine_chunk_destinations test 1d9204ac4 Use remote frag index to calculate suffix diff 5b5ed29ab py3: object server 3a8f5dbf9 Verify client input for v4 signatures 1a51604b2 s3api: Look for more indications of aws-chunked uploads b35fc4118 s3api: Delete multipart uploads via multi-delete 37693a4e1 Run ceph-s3-tests job less 0d29b01d2 py3: Port the acl, account_quotas, cname_lookup, container_sync 9856d6eb2 Add a comment about the module list for py3 in tox.ini 0f505ad96 py3: adapt the account server completely 02841ee0c fix documentation of default fcccb52ff s3api: Stop relying on container listings during multipart uploads de1c4f0d5 add an upgrade job from rocky 014511d31 s3api: Stop mangling Authorization header for v4 signatures 284bbdd39 Add slo_manifest_hook callback 43412c73d Treat all invalid frag indexes the same 53ae01b9c Modify the description content 39f6d5a5f Py3: Fix two unbound variables a563ba26f Implemented the fix to handle the HTTP request methods other than GET. Diffstat (except docs and test files) ------------------------------------- .mailmap | 2 +- .zuul.yaml | 37 +- AUTHORS | 6 +- CHANGELOG | 100 ++ api-ref/source/parameters.yaml | 2 +- etc/object-server.conf-sample | 9 + etc/proxy-server.conf-sample | 3 +- .../notes/2_15_2_release-6996eccabba558b4.yaml | 22 + .../notes/2_17_1_release-dd6e6879cbb94f85.yaml | 11 + .../notes/2_19_1_release-5072dd72557f5708.yaml | 16 + .../notes/2_21_0_release-d8ae33ef18b7be3a.yaml | 69 ++ swift/account/reaper.py | 90 +- swift/account/server.py | 23 +- swift/account/utils.py | 8 +- swift/cli/info.py | 14 + swift/cli/manage_shard_ranges.py | 24 +- swift/cli/recon.py | 5 +- swift/common/bufferedhttp.py | 8 + swift/common/constraints.py | 7 +- swift/common/container_sync_realms.py | 16 +- swift/common/db.py | 13 +- swift/common/db_replicator.py | 5 +- swift/common/exceptions.py | 4 + swift/common/middleware/bulk.py | 24 +- swift/common/middleware/copy.py | 19 +- swift/common/middleware/crypto/crypto_utils.py | 10 +- swift/common/middleware/crypto/keymaster.py | 59 +- swift/common/middleware/crypto/kmip_keymaster.py | 19 +- swift/common/middleware/dlo.py | 3 +- swift/common/middleware/formpost.py | 34 +- swift/common/middleware/proxy_logging.py | 2 + .../middleware/s3api/controllers/multi_delete.py | 26 +- .../middleware/s3api/controllers/multi_upload.py | 70 +- .../common/middleware/s3api/controllers/s3_acl.py | 2 +- swift/common/middleware/s3api/s3request.py | 75 +- swift/common/middleware/s3api/s3token.py | 19 + swift/common/middleware/slo.py | 19 +- swift/common/middleware/tempauth.py | 5 +- swift/common/middleware/versioned_writes.py | 58 +- swift/common/request_helpers.py | 46 +- swift/common/ring/ring.py | 68 +- swift/common/swob.py | 81 +- swift/common/utils.py | 78 +- swift/common/wsgi.py | 45 +- swift/container/backend.py | 20 +- swift/container/reconciler.py | 18 +- swift/container/replicator.py | 21 +- swift/container/server.py | 18 +- swift/container/sharder.py | 22 +- swift/obj/diskfile.py | 192 ++-- swift/obj/mem_diskfile.py | 3 + swift/obj/reconstructor.py | 166 +-- swift/obj/replicator.py | 6 +- swift/obj/server.py | 22 +- swift/obj/ssync_receiver.py | 19 +- swift/obj/ssync_sender.py | 23 +- swift/proxy/controllers/account.py | 6 +- swift/proxy/controllers/base.py | 79 +- swift/proxy/controllers/container.py | 19 +- swift/proxy/controllers/obj.py | 63 +- swift/proxy/server.py | 2 +- test/functional/__init__.py | 25 +- test/functional/s3api/test_multi_upload.py | 32 +- test/functional/swift_test_client.py | 168 ++-- test/functional/test_container.py | 41 +- test/functional/test_dlo.py | 5 +- test/functional/test_object.py | 34 +- test/functional/test_slo.py | 13 + test/functional/test_symlink.py | 7 +- test/probe/test_reconstructor_rebuild.py | 66 +- test/probe/test_replication_servers_working.py | 37 +- test/probe/test_sharder.py | 4 +- test/unit/__init__.py | 83 +- test/unit/account/test_reaper.py | 21 +- test/unit/account/test_server.py | 77 +- test/unit/cli/test_info.py | 145 +++ test/unit/cli/test_manage_shard_ranges.py | 67 +- .../common/middleware/crypto/test_encryption.py | 2 +- .../common/middleware/crypto/test_keymaster.py | 176 +++- .../middleware/crypto/test_kmip_keymaster.py | 58 +- test/unit/common/middleware/s3api/__init__.py | 5 +- test/unit/common/middleware/s3api/helpers.py | 5 +- .../common/middleware/s3api/test_multi_delete.py | 45 +- .../common/middleware/s3api/test_multi_upload.py | 171 ++-- test/unit/common/middleware/s3api/test_obj.py | 127 ++- test/unit/common/middleware/s3api/test_s3api.py | 31 +- .../unit/common/middleware/s3api/test_s3request.py | 80 +- test/unit/common/middleware/test_account_quotas.py | 6 +- test/unit/common/middleware/test_cname_lookup.py | 50 +- test/unit/common/middleware/test_container_sync.py | 34 +- test/unit/common/middleware/test_copy.py | 46 +- test/unit/common/middleware/test_formpost.py | 514 ++++++---- test/unit/common/middleware/test_name_check.py | 14 +- test/unit/common/middleware/test_quotas.py | 16 +- test/unit/common/middleware/test_ratelimit.py | 28 +- test/unit/common/middleware/test_read_only.py | 23 +- test/unit/common/middleware/test_recon.py | 68 +- test/unit/common/middleware/test_slo.py | 53 + test/unit/common/middleware/test_tempauth.py | 30 + .../common/middleware/test_versioned_writes.py | 2 +- test/unit/common/ring/test_ring.py | 4 + test/unit/common/test_container_sync_realms.py | 17 +- test/unit/common/test_db_replicator.py | 2 +- test/unit/common/test_swob.py | 18 + test/unit/common/test_utils.py | 86 +- test/unit/common/test_wsgi.py | 51 - test/unit/container/test_auditor.py | 12 +- test/unit/container/test_backend.py | 83 +- test/unit/container/test_reconciler.py | 22 +- test/unit/container/test_server.py | 194 ++-- test/unit/container/test_sharder.py | 73 +- test/unit/container/test_sync.py | 37 +- test/unit/container/test_updater.py | 47 +- test/unit/helpers.py | 34 +- test/unit/obj/common.py | 6 +- test/unit/obj/test_auditor.py | 44 +- test/unit/obj/test_diskfile.py | 164 +-- test/unit/obj/test_expirer.py | 3 +- test/unit/obj/test_reconstructor.py | 1058 ++++++++++++++------ test/unit/obj/test_replicator.py | 42 +- test/unit/obj/test_server.py | 839 ++++++++-------- test/unit/obj/test_ssync.py | 29 +- test/unit/obj/test_ssync_receiver.py | 35 +- test/unit/obj/test_ssync_sender.py | 44 +- test/unit/obj/test_updater.py | 24 +- test/unit/proxy/controllers/test_base.py | 80 +- test/unit/proxy/controllers/test_container.py | 5 +- test/unit/proxy/controllers/test_obj.py | 443 ++++---- test/unit/proxy/test_mem_server.py | 6 + test/unit/proxy/test_server.py | 286 ++++-- test/unit/proxy/test_sysmeta.py | 3 +- tox.ini | 51 +- 137 files changed, 5698 insertions(+), 3218 deletions(-)