We eagerly announce the release of: swift 2.37.0 This release is part of the gazpacho release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.37.0 ^^^^^^ New Features ************ * The s3token middleware now passes service auth tokens to Keystone if credentials are provided. This is required to enable S3 API access for Keystone users when using Keystone >25.0.0, !=26.0.0, !=26.0.1, !=27.0.0, !=28.0.0. See etc/proxy-server.conf-sample for configuration details. For more information, see OSSA-2025-002 and bug #2119646. * The s3token middleware now caches credential secrets for one minute by default, if credentials are provided. Secret-caching typically reduces the load on Keystone and is required for Keystone users to be able to use signed aws-chunked transfers. To return to prior behavior, explicitly set "secret_cache_duration = 0" in the "[filter:s3api]" section of your proxy-server.conf. * The KMS keymaster now supports selecting the endpoint returned in the Keystone catalog via the "barbican_region_name" configuration option. This may be useful in multi-region deployments which have multiple endpoints. * The request line-length limit is now configurable for all WSGI servers via the "max_request_line" option in the "[swift- constraints]" section of swift.conf. By default, continue to use eventlet's default of 8192 bytes. * The following new metrics were added when using labeled metrics: * The proxy-logging middleware may now emit real-time transfer metrics. See the "statsd_emit_buffer_xfer_bytes_seconds" option in etc/proxy-server.conf-sample for more information. * The proxy-logging middleware now includes an "api" label whose value may be "swift" or "S3" depending on whether the client request is serviced by the swift API or S3 API. * The s3api middleware now emits a counter recording the usage of various protocol-related headers. * The container-sharder now emits a timing metric for the length of time between shard range creation and cleaving. * "swift-manage-shard-ranges" now defaults to committing pending updates before looking for shard range boundaries. A new option, "-- skip-commits", may be used to restore previous behavior. * Added a "--clobber-hardlink-collisions" option to "swift-object- relinker". With this option enabled during the relink phase the relinker will quarantine the colliding file in the new target part dir and retry the relink. During the cleanup phase it will ignore the un-matched inode "collision" and allow the cleanup of the old file in the old part dir similar to tombstones. Upgrade Notes ************* * Removed fallback support using netifaces; "getifaddrs" is now always used to determine available IP addresses. Bug Fixes ********* * Improved checksum validation for S3 API DeleteObjects requests. * POST requests are more likely to receive a 503 response in the face of backend inconsistencies. * Writes to sharded containers are less likely to have their updates sent to the root container. This uses a new cooperative-token mechanism to limit the number of concurrent shard range queries to the root container; see the "[app:proxy-server]" section of etc /proxy-server.conf-sample for configuration options. * Fixed the "swift_dir" option for WSGI servers; the file "/etc/swift/swift.conf" no longer needs to exist when that option is set. * Fixed an object-server error when there is a part-power increase in progress and there was an issue marking the file in the new partition space as durable. * Device names are now included in sharded database IDs, similar to regular databases. This provides more context when examining incoming/outgoing sync tables or sharding CleaveContexts. * Database replicators now clean up temporary files older than "reclaim_age". * Various other minor bug fixes and improvements. Changes in swift 2.36.0..2.37.0 ------------------------------- 555026d20 AUTHORS/CHANGELOG for 2.37.0 fb6b7095a Add barbican_region_name support to Swift kms_keymaster middleware ef0d8591c CI: Make swift-upload-image non-voting again ee682f4c2 CI: Update dockerhub secret yet again 1b0c11b68 Updating the dockerhub secret (again^2) 9b5236339 CI: Update dockerhub secret (again) a572a08b3 Fix flakey proxy server test_node_timing ce829c45c CI: clean up old swift-*-image-py3 jobs 732fd859d Fix flakey db_replicator unit test 64bde7e05 proxy-logging: Add 'api' labels 0862c231a Recliam db_dir/*.tmp files 86d1dee2a Update dockerhub secret 1971983de test_backend.py: refactor TestUpdateNewItemFromExisting db13f7116 probe tests: fix timestamp handling 949f0fdac tests: assert that X-Timestamp headers are valid 8f82e4586 test_sync.py: fix timestamp handling 327401e18 docs: Clarify circumstances for using clobber_hardlink_collisions 15a1c0287 Update rolling-upgrade jobs f84486e96 versioned_writes: use new parse_date_header helper 50df63daa tests: relocate timestamp tests to test_timestamp.py 2e8d27607 Remove duplicate sample config option 9366678a9 legacy versioning: use Timestamp to create marker name 977b96b89 test_object_versioning.py: used mocked Timestamps 326fad040 cli/info.py: use Timestamp.zero as default timestamp c36e789cd proxy/test_server.py: test_policy_IO_override timestamp refactoring. 3fea9f474 Replace deprecated warn method 55eddb9c6 container/test_server: cleanup Timestamp usage 1698fc324 OldestAsyncPendingTracker accepts Timestamp.internal bf1e370b2 test_relinker.py: fix delta timestamp setup 0ceff0fd5 test_base.py: tighten up use of Timestamp 768815b4b max_request_line: add probe test assertions a6c3e7dd6 wsgi: Add url length limit config 01d7eef88 Add labeled metrics to s3api 9abd8ae71 sharder: use correct Timestamp formats 80bbcfeb9 test_utils: cleanup Timestamp usage 5df4e758f slo: use Timestamp.zero() as default in RespAttrs db578c107 Add a Timestamp.zero() method 3123422cd test_relinker: cleanup Timestamp usage 6dccca38c container tests: Use TS.normal for db_names b76c733be test: reconciler tests to use Timestamp class over time 259afd2a7 ssync: Refactor tests to use Timestamp and new assert df5131b8f Device in fresh db id suffix d228a7323 Adds labeled cleaving metrics to the sharder 9babfefad Enhances sharding_in_progress metrics in recon log 22605d21e cli/test_info.py: cleanup Timestamp usage ca9db701c account/test_reaper.py: cleanup Timestamp usage e6f1b06c9 test/unit/container: cleanup Timestamp usage a70e33e73 test_constraints.py: cleanup Timestamp usage 91dd65d36 test_symlink.py: cleanup Timestamp usage 0190d81e4 test_diskfile.py: fix Timestamp usage 254100ef1 tests: container/test_backend.py cleanup 5658355bf account/test_utils: cleanup Timestamp usage 18b861bd6 obj test_expirer.py and test_auditor.py: fix timestamp usage 800cf3211 test_reconstructor.py: make hashes assertions less brittle a07946329 test_diskfile.py: fix failing test if O_TMPFILE unsupported d31b201d6 test_db.py: clean up usage of Timestamps 760829a75 account: Use Timestamps better in test_backend unit tests 48937cd18 TombstoneReclaimer: quote string timestamp in sql query c8da8676f tests: clarify timestamp formats 02bc7e448 tests: pass Timestamps to date_header_format 98005fad9 Allow Timestamps in swob's date-header helpers 41376fca5 trivial: Use swob date-header helpers more 96e7db0b3 swob: add a date_header_format helper function affdbf71a trivial: tests: add failure context to flakey assertion cbc3b42d7 s3api: verify checksums on DeleteObjects POSTs 5f89d14eb s3token: Enable secret caching by default ee6f4c6a0 trivial: tests: clarify exception handling ef8b2a5f3 Return 503 for POST request with mixed 202/404 responses e7bb2a385 s3token: Pass service auth token to Keystone a785025b2 reno: Update master for unmaintained/2024.1 dcd5a265f proxy-logging: Add real-time transfer bytes counters e450b8c67 tests: belts and braces idiomatic patching 89e9ced9e tests: idiomatic ring patching f256bc7eb tests: remove some global patching 8cbe10552 Update resource type in labels.html 7744e4a43 Move Pete Zaitcev to Core Emeritus 9d7e7e27a Provide some s3 helper methods for other middlewares to use. fac55ced3 test-db-replicator (trivial): just one tmpdir c0fefe80b trivial test_[db_]replicator cleanup be62933d0 relinker: allow clobber-hardlink-collision 41bf72a5c common.db_replicator: log container, db paths consistently 5568dd09b Fix swift_dir setting in WSGI servers 97e00e208 trivial: reorder AccountBroker.path methods 3c6e967a5 test: fix AccountBroker.path tests a6bde729c Test each method in test_crossdomain_get_only 64bb04139 Assert metadata of SLO PUT from container sync 79feb12b2 docs: More proxy-server.conf-sample cleanup b55f13c75 test: move import to top of file 389747a8b doc: specify seconds in proxy-server.conf-sample ba1ab9d11 Adds --skip-commits to s-m-s-r d9883d083 proxy: use cooperative tokens to coalesce updating shard range requests into backend d353f15fa account-broker: add resilient path property with lazy cache 707a65ab3 common: add memcached based cooperative token mechanism. b035ed138 add Christian O to AUTHORS 7b05356bd test: do not create timestamp collision unnecessarily 815393dff test: fix module state pollution c26c7b8ed tests: simplify TestGlobalSetupObjectReconstructor setUp bd27fc6ba cleaning up and fixing some links 63eeb005b Update master for stable/2025.2 397f94c73 diskfile: Fix UnboundLocalError during part power increase b5e6964a2 s3api: fix test_service with pre-existing buckets 484672b7d Fully remove netifaces Diffstat (except docs and test files) ------------------------------------- .mailmap | 2 + .zuul.yaml | 43 +- AUTHORS | 5 +- CHANGELOG | 86 + CONTRIBUTING.rst | 29 +- etc/keymaster.conf-sample | 8 +- etc/object-server.conf-sample | 11 + etc/proxy-server.conf-sample | 103 +- etc/swift.conf-sample | 11 + .../notes/release-2.37.0-7a89cc30f85f03e0.yaml | 102 ++ releasenotes/source/2024.1.rst | 2 +- releasenotes/source/2025.2.rst | 6 + releasenotes/source/index.rst | 1 + roles/additional-keystone-users/tasks/main.yaml | 35 + roles/dsvm-additional-middlewares/tasks/main.yaml | 76 +- swift/account/backend.py | 23 +- swift/cli/info.py | 2 +- swift/cli/manage_shard_ranges.py | 31 +- swift/cli/relinker.py | 177 +- swift/common/constraints.py | 2 + swift/common/container_sync_realms.py | 2 +- swift/common/db.py | 38 +- swift/common/db_replicator.py | 176 +- swift/common/exceptions.py | 6 +- swift/common/middleware/crypto/kms_keymaster.py | 9 +- swift/common/middleware/proxy_logging.py | 177 +- .../middleware/s3api/controllers/multi_delete.py | 7 +- .../middleware/s3api/controllers/multi_upload.py | 23 +- swift/common/middleware/s3api/exception.py | 12 + swift/common/middleware/s3api/s3api.py | 118 +- swift/common/middleware/s3api/s3request.py | 107 +- swift/common/middleware/s3api/s3token.py | 65 +- swift/common/middleware/s3api/utils.py | 161 ++ swift/common/middleware/slo.py | 5 +- swift/common/middleware/tempurl.py | 10 +- swift/common/middleware/versioned_writes/legacy.py | 18 +- .../versioned_writes/object_versioning.py | 14 +- swift/common/swob.py | 51 +- swift/common/utils/__init__.py | 257 ++- swift/common/utils/config.py | 15 + swift/common/utils/ipaddrs.py | 210 +-- swift/common/utils/timestamp.py | 10 + swift/common/wsgi.py | 10 +- swift/container/backend.py | 23 +- swift/container/reconciler.py | 10 +- swift/container/replicator.py | 47 +- swift/container/server.py | 20 +- swift/container/sharder.py | 435 +++-- swift/container/sync.py | 6 +- swift/obj/diskfile.py | 17 +- swift/obj/server.py | 12 +- swift/obj/updater.py | 7 +- swift/proxy/controllers/base.py | 146 +- swift/proxy/controllers/container.py | 4 +- swift/proxy/controllers/obj.py | 210 ++- swift/proxy/server.py | 23 +- test/__init__.py | 13 + test/debug_logger.py | 8 + test/probe/brain.py | 18 + test/probe/common.py | 6 +- test/probe/test_container_failures.py | 3 +- test/probe/test_object_expirer.py | 7 +- test/probe/test_object_handoff.py | 72 + test/probe/test_sharder.py | 12 +- test/probe/test_signals.py | 79 +- test/s3api/__init__.py | 13 +- test/s3api/test_input_errors.py | 267 ++- test/s3api/test_service.py | 91 +- test/sample.conf | 4 +- test/unit/__init__.py | 70 + test/unit/account/test_backend.py | 119 +- test/unit/account/test_reaper.py | 21 +- test/unit/account/test_replicator.py | 7 +- test/unit/account/test_utils.py | 6 +- test/unit/cli/test_info.py | 81 +- test/unit/cli/test_manage_shard_ranges.py | 49 +- test/unit/cli/test_relinker.py | 328 +++- test/unit/common/middleware/helpers.py | 14 +- test/unit/common/middleware/s3api/__init__.py | 10 +- test/unit/common/middleware/s3api/test_bucket.py | 18 + .../common/middleware/s3api/test_multi_delete.py | 31 +- .../common/middleware/s3api/test_multi_upload.py | 11 +- test/unit/common/middleware/s3api/test_obj.py | 31 +- test/unit/common/middleware/s3api/test_s3api.py | 573 ++++++- .../unit/common/middleware/s3api/test_s3request.py | 112 ++ test/unit/common/middleware/s3api/test_s3token.py | 15 + test/unit/common/middleware/s3api/test_utils.py | 203 +++ test/unit/common/middleware/test_container_sync.py | 19 + test/unit/common/middleware/test_crossdomain.py | 4 +- .../common/middleware/test_object_versioning.py | 233 +-- test/unit/common/middleware/test_proxy_logging.py | 1750 +++++++++++++++++++- test/unit/common/middleware/test_slo.py | 34 +- test/unit/common/middleware/test_symlink.py | 6 +- .../common/middleware/test_versioned_writes.py | 65 +- test/unit/common/test_constraints.py | 5 +- test/unit/common/test_db.py | 179 +- test/unit/common/test_db_replicator.py | 521 ++++-- test/unit/common/test_swob.py | 17 + test/unit/common/test_utils.py | 1065 ++++++++++-- test/unit/common/test_wsgi.py | 98 +- test/unit/common/utils/test_config.py | 27 + test/unit/common/utils/test_ipaddrs.py | 38 - test/unit/common/utils/test_timestamp.py | 155 ++ test/unit/container/test_backend.py | 467 +++--- test/unit/container/test_reconciler.py | 234 +-- test/unit/container/test_replicator.py | 291 +++- test/unit/container/test_server.py | 224 ++- test/unit/container/test_sharder.py | 384 +++-- test/unit/container/test_sync.py | 28 +- test/unit/container/test_updater.py | 12 +- test/unit/obj/common.py | 8 + test/unit/obj/test_auditor.py | 25 +- test/unit/obj/test_diskfile.py | 303 +++- test/unit/obj/test_expirer.py | 8 +- test/unit/obj/test_reconstructor.py | 234 ++- test/unit/obj/test_server.py | 608 ++++--- test/unit/obj/test_ssync_sender.py | 464 +++--- test/unit/obj/test_updater.py | 68 +- test/unit/proxy/controllers/test_base.py | 76 +- test/unit/proxy/controllers/test_obj.py | 1076 +++++++++++- test/unit/proxy/test_server.py | 304 +++- 124 files changed, 11359 insertions(+), 3215 deletions(-)