We are satisfied to announce the release of: swift 2.33.0: OpenStack Object Storage This release is part of the caracal release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.33.0 ^^^^^^ New Features ************ * Prefix-based tempurls may now be used to explore staticweb listings within that prefix. Note that this opens a new ability to list containers from tempurls, but only if staticweb's "X-Container- Meta- Web-Listings" is enabled. * When generating index pages from listings, staticweb now sends an HTML5 doctype. This makes them PEP 503 compliant, allowing their continued use for simple Python package repositories. * Add basic read support for S3 object locking. This improves compatibility with an Ansible S3 module. Write support is not yet implemented, so get-object-lock-configuration will always 404. * Added a "swift-reload" command to assist with safely reloading WSGI servers. * Daemons now send "STOPPING" and "RELOADING" systemd notifications when the service is configured with "Type=notify". * Added more metrics to the container-server, allowing GET and PUT timings to be broken out for listings, shard range operations, and container creation. * Added a counter metric to the proxy server when caching shard ranges. * "swift-account-info" and "swift-container-info" now accept a "-- sync" flag to show information from the incoming/outgoing sync tables. * Several fixes to prepare for Python 3.12 support. While not yet tested in the gate, initial manual testing looks promising. * Added support for recent versions of eventlet. Bug Fixes ********* * S3 API fixes: * When the "+segments" container's storage policy differs from that of the primary container, completed manifests are now written with the correct policy in the primary container. * If there's a conflict deleting the in-progress-upload marker when completing a multipart-upload, a 503 is now returned to the client, prompting it to retry. * Added "Accept-Ranges: bytes" to object responses. Range requests have always been supported; now, that support is properly advertised. * Static large object fixes: * Fixed a server error when handling conditional GET requests. * Return an error if the SLO manifest could not be parsed. Previously, a zero-byte response was returned. * Proxy server fixes: * Added a new "swift.proxy_logging_status" request environment key that middlewares may use to override the logged status for a request. * Transaction IDs are included in more error responses. * The "recoverable_node_timeout" option no longer applies to "X-Newest" GET requests. * Improved error-handling in multi-part range responses. * Sharding fixes: * Prevent resets of a shard range's epoch. * Cleaned up "X-Backend-*" headers in listing responses. * Reduced the frequency of "Reclaimable db stuck waiting for shrinking" messages when a root DB has been deleted but its shards have not been shrunk away. * The more-efficient shard range structure from the last release is now used when fetching ranges from the backend. * Include more information in shard-replication warnings. * Object server fixes: * Object POSTs and chunked PUTs are no longer accepted when the target drive is already past its "fallocate_reserve". DELETEs are still allowed. * Added the ability to configure cooperative yielding when servicing GET responses, via the "cooperative_period" option. See the example config for more information. * Invalid "hashes.invalid" entries are now ignored, rather than causing a complete partition rehash. * Per-service "auto_create_account_prefix" configuration options have been removed. These options were deprecated in favor of "swift.conf" configuration in Swift 2.24.0, part of the OpenStack Ussuri release. * Daemons send object updates via the replication network in more cases. * The dark-data object audit watcher now works with sharded containers. Previously, it would think that all data files were absent from listings. * Various other minor bug fixes and improvements. Changes in swift 2.32.0..2.33.0 ------------------------------- 5f95b6b26 AUTHORS/CHANGELOG for 2.33.0 9279e3d2c CI: Bring unit test jobs in line with 2024.1 tested runtimes b06ffea94 CI: Update rolling-upgrade jobs to point to unmaintained branches dc379af89 reno: Update master for unmaintained/xena 3ad1565f0 reno: Update master for unmaintained/wallaby 0d8c89b12 reno: Update master for unmaintained/victoria 6a426f7fa sharder: Add periodic_warnings_interval to example config cc2778004 docs: add discussion of content-type metadata 4135133a6 memcachering: change failed to yield log message bd3b2256a CI: Remove centos-7 jobs 275af9e00 CI: fix rolling-upgrade jobs f32f2dd02 CI: pin python-dateutil for py2 130188b6c zero bytes manifests are not legacy 1ee9b1e3b tests: Clear txn id on init for all debug loggers d5877179a Object-server: add periodic greenthread yielding during file read. 2da150b89 Fix diskfile test failing on macOS 2500fbeea proxy: don't use recoverable_node_timeout with x-newest 8061dfb1c proxy-server: de-duplicate _get_next_response_part method 89dd51531 test: all primary error limit is error c522f5676 Add ClosingIterator class; be more explicit about closes db2caea1f reno: Update master for unmaintained/yoga ce9e56a6d lint: Consistently use assertIsInstance 76ca11773 lint: Up-rev hacking 8227f4539 sharding: don't replace own_shard_range without an epoch c5d743347 diskfile: Ignore invalid suffixes in invalidations file 1936f6735 replicator: Rename update_deleted to revert 8c4e65a6b staticweb: Work with prefix-based tempurls e96a08102 tests: Fix float expectations for py312 dc3eda7e8 proxy: don't send multi-part terminator when no parts sent 252f0d36b proxy: only use listing shards cache for 'auto' listings 0a6daa1ad test: swift.proxy_logging_status is really lazy (in a good way!) a16e1f55a Improve unit tests for proxy GET ChunkReadTimeouts b1836f936 Update malformed_example.db to actually be malformed 0d60af250 Prevent installation of known-broken eventlet 5c4176aaf Only try to install py2 dev libraries for py2 jobs a736bd96f CI: Treat grenade-skip-level jobs like grenade jobs e39078135 tests: Exercise recent eventlet breakage without XFS 7e3925aa9 tests: Fix probe test when encryption is enabled 3ab9e45d6 Work with latest eventlet (again) 52c80d652 cli: add --sync to db info to show syncs 03b66c94f Proxy: Use namespaces when getting listing/updating shards c07393338 Container-server: add container namespaces GET f3a32367b ContainerBroker.get_shard_ranges(): states must be a list 6b9133429 Make the dark data watcher work with sharded containers f2c6c1941 container-server unit tests: use self.ts everywhere ac7eb8ac9 staticweb: Generate HTML5 pages 3cbf01b12 Update python classifier in setup.cfg 603122a32 Stop using deprecated datetime.utc* functions bf7f3ff2f tests: Switch get_v4_amz_date_header to take timedeltas fe0d138ea Get tests passing with latest eventlet bd64748a0 Document allowed_digests for formpost middleware 5af7719ef Support swift.proxy_logging_status in request env 365db2027 FakeSwift: use HTTPMethodNotAllowed not HTTPNotImplemented b07d87c4b tests: use subclasses for S3Acl tests 1c31973d3 test: couple raw manifests with their TestCase bcb881088 tests: consolidate Namespace/ShardRange _check_name 71ad062bc proxy: remove x-backend-record-type=shard in object listing 93f812a51 Add account and container properties to Namespace 7dadc5cac test: use debug_logger instead of Mock logger 36ce42f54 Relocate GetOrHeadHandler unit tests 63defd143 s3api: return 503 if mpu complete gets 409 deleting marker e375ce5bf replication network aware helper for container-updater in proxy a10205560 Container-server: add request timing metrics for PUT/GET object/shard/container. 6f890d2ba proxy: move _get_shard_ranges to ObjectController 49b19613d Remove per-service auto_create_account_prefix 72ac5b3be proxy: refactor to share namespace cache helpers 60c04f116 s3api: Stop propagating storage policy to sub-requests 38e645103 tests: FakeSwift._responses is always a list of tuples 4a37a2976 slo: refactor GET/HEAD response handling c94e1c151 docs: Document WSGI server process management 736369878 Add mailmap entry for my new mail d0d553394 Utils: fix Namespace and ShardRange attribute encoding in py2. 3dab88bdf tests: refactor SLO size/etag sysmeta tests 965908d75 proxy: emit metric when updating namespaces memcache set fails 2b8036447 sharder: improve failed shard replication warning logs 53ccd0264 Prevent spurious ChunkReadTimeout during test runner exit f14b1de4d sharder: reduce rate of 'Reclaimable db...' warnings 55f7833d8 systemd: Send STOPPING/RELOADING notifications 212525118 Add a swift-reload command fcf1110ab CI: Fix some known-failure formatting 20ff64215 stats: Round timings at 4 decimal places 0893cedc3 Include accept-ranges header in s3api response 002ad59c9 tests: drop pkg_resources from test_auditor.py 0996433fe s3api: Add basic GET object-lock support d31a54a65 object: Block POSTs and chunked PUTs when already past reserve 667ee6e24 Swap find_executable for which 01be00044 tests: Stop using distutils.dir_util.mkpath 3f3f5be9b tests: boto is always <3.0 f871591ba tests: swiftclient supports insecure bd5076ebe Fix repeated unnecessarily 'os.makedirs()' in test_recon_cron.py 62a9dbca7 Add unittest of swift-recon-cron f8c94d6bb proxy-server: add replicated GET path tests 4051053be diskfile: Pass extension when creating DiskFileWriters 4073eacf6 CI: Stop collecting coverage reports for py311 5392a2057 tests: Add test(s) for MPU part copy from range 0ffc7cab8 Fix a typo in common.utils.__init__ 7d68b037d It's not all statuses None 370c15c03 Add rolling upgrade job coming from bobcat ff4e1b4fd Update master for stable/2023.2 451ae26a8 Improve FakeSwift Backend-Ignore-Range support 4a80963b4 debug_logger: ensure NOTICE level is registered 68b71752a s3api: use string for swift info request method 7b39698d0 wsgi: bad request syntax response missing txn-id 25e643eb8 tests: Include filter with no _pipeline_final_app 9f385c07f proxy: Get rid of MetricsPrefixLoggerAdapter 8a2c8b128 Update doc for include_service_catalog with app cred 01c7ade79 proxy-server: fix node error limiting 6e5b5a659 slo: 500 if we can't load the manifest 91a6b52a4 tests: Get test_handoff_non_durable passing with encryption enabled Diffstat (except docs and test files) ------------------------------------- .mailmap | 3 +- .zuul.yaml | 102 +- AUTHORS | 4 +- CHANGELOG | 110 +- bin/swift-container-info | 3 + bin/swift-recon-cron | 62 +- bindep.txt | 4 +- etc/container-server.conf-sample | 6 + etc/object-server.conf-sample | 6 + etc/proxy-server.conf-sample | 4 + lower-constraints.txt | 1 + py2-constraints.txt | 4 + .../notes/3_33_0_release-d208917f5012cedd.yaml | 129 + releasenotes/source/2023.2.rst | 6 + releasenotes/source/index.rst | 1 + releasenotes/source/victoria.rst | 2 +- releasenotes/source/wallaby.rst | 2 +- releasenotes/source/xena.rst | 2 +- releasenotes/source/yoga.rst | 2 +- requirements.txt | 2 +- setup.cfg | 2 + swift/account/server.py | 12 +- swift/cli/info.py | 29 +- swift/cli/recon_cron.py | 73 + swift/cli/reload.py | 141 + swift/common/daemon.py | 14 +- swift/common/db.py | 20 +- swift/common/http_protocol.py | 79 +- swift/common/manager.py | 23 +- swift/common/memcached.py | 30 +- swift/common/middleware/proxy_logging.py | 62 +- swift/common/middleware/s3api/acl_handlers.py | 2 +- .../middleware/s3api/controllers/__init__.py | 3 + swift/common/middleware/s3api/controllers/acl.py | 5 +- .../common/middleware/s3api/controllers/logging.py | 4 +- .../middleware/s3api/controllers/multi_upload.py | 36 +- .../middleware/s3api/controllers/object_lock.py | 44 + swift/common/middleware/s3api/s3api.py | 4 + swift/common/middleware/s3api/s3request.py | 31 +- swift/common/middleware/s3api/s3response.py | 15 +- swift/common/middleware/s3api/utils.py | 2 +- swift/common/middleware/slo.py | 693 ++-- swift/common/middleware/staticweb.py | 59 +- swift/common/middleware/tempurl.py | 223 +- swift/common/request_helpers.py | 40 +- swift/common/storage_policy.py | 2 +- swift/common/swob.py | 7 +- swift/common/utils/__init__.py | 365 +- swift/common/utils/timestamp.py | 25 +- swift/common/wsgi.py | 8 + swift/container/backend.py | 147 +- swift/container/replicator.py | 48 +- swift/container/server.py | 409 ++- swift/container/sharder.py | 80 +- swift/container/updater.py | 1 + swift/obj/auditor.py | 3 +- swift/obj/diskfile.py | 74 +- swift/obj/expirer.py | 14 +- swift/obj/mem_diskfile.py | 3 +- swift/obj/mem_server.py | 1 + swift/obj/replicator.py | 4 +- swift/obj/server.py | 35 +- swift/obj/updater.py | 1 + swift/obj/watchers/dark_data.py | 106 +- swift/proxy/controllers/account.py | 5 +- swift/proxy/controllers/base.py | 344 +- swift/proxy/controllers/container.py | 563 ++-- swift/proxy/controllers/obj.py | 239 +- swift/proxy/server.py | 14 +- test-requirements.txt | 2 +- test/debug_logger.py | 12 +- test/functional/__init__.py | 18 +- test/functional/s3api/__init__.py | 28 + test/functional/s3api/test_bucket.py | 54 +- test/functional/s3api/test_multi_upload.py | 179 +- test/functional/s3api/test_object.py | 81 +- test/functional/test_protocol.py | 53 + test/functional/test_staticweb.py | 214 ++ test/functional/test_tempurl.py | 41 +- test/probe/test_mixed_policy_mpu.py | 157 + test/probe/test_object_versioning.py | 20 +- test/probe/test_reconstructor_revert.py | 18 +- test/probe/test_sharder.py | 650 +++- test/s3api/test_object_lock.py | 51 + test/unit/__init__.py | 116 +- test/unit/account/test_server.py | 8 - test/unit/cli/test_info.py | 36 +- test/unit/cli/test_manage_shard_ranges.py | 2 +- test/unit/cli/test_recon.py | 2 +- test/unit/cli/test_recon_cron.py | 50 + test/unit/cli/test_reload.py | 220 ++ test/unit/common/malformed_example.db | Bin 8192 -> 8192 bytes test/unit/common/middleware/helpers.py | 132 +- test/unit/common/middleware/s3api/__init__.py | 172 +- test/unit/common/middleware/s3api/helpers.py | 81 - test/unit/common/middleware/s3api/test_acl.py | 152 +- test/unit/common/middleware/s3api/test_bucket.py | 893 ++--- test/unit/common/middleware/s3api/test_etree.py | 2 +- test/unit/common/middleware/s3api/test_helpers.py | 69 - .../common/middleware/s3api/test_multi_delete.py | 186 +- .../common/middleware/s3api/test_multi_upload.py | 1733 +++++----- test/unit/common/middleware/s3api/test_obj.py | 1202 +++---- .../common/middleware/s3api/test_object_lock.py | 67 + test/unit/common/middleware/s3api/test_s3_acl.py | 181 +- test/unit/common/middleware/s3api/test_s3api.py | 16 +- .../unit/common/middleware/s3api/test_s3request.py | 17 +- test/unit/common/middleware/s3api/test_service.py | 40 +- .../common/middleware/s3api/test_subresource.py | 11 +- test/unit/common/middleware/test_helpers.py | 375 ++- test/unit/common/middleware/test_memcache.py | 2 +- test/unit/common/middleware/test_proxy_logging.py | 193 +- test/unit/common/middleware/test_slo.py | 3550 ++++++++++++++------ test/unit/common/middleware/test_staticweb.py | 66 + test/unit/common/middleware/test_symlink.py | 2 - test/unit/common/middleware/test_tempurl.py | 103 +- test/unit/common/ring/test_builder.py | 49 +- test/unit/common/test_daemon.py | 2 +- test/unit/common/test_db.py | 51 + test/unit/common/test_db_auditor.py | 2 +- test/unit/common/test_db_replicator.py | 4 +- test/unit/common/test_exceptions.py | 2 +- test/unit/common/test_http_protocol.py | 116 +- test/unit/common/test_internal_client.py | 28 +- test/unit/common/test_manager.py | 8 +- test/unit/common/test_memcached.py | 64 +- test/unit/common/test_request_helpers.py | 108 +- test/unit/common/test_storage_policy.py | 2 +- test/unit/common/test_swob.py | 69 +- test/unit/common/test_utils.py | 583 +++- test/unit/common/test_wsgi.py | 51 +- test/unit/container/test_backend.py | 688 ++-- test/unit/container/test_reconciler.py | 7 +- test/unit/container/test_replicator.py | 125 + test/unit/container/test_server.py | 906 ++++- test/unit/container/test_sharder.py | 149 +- test/unit/container/test_sync.py | 2 +- test/unit/helpers.py | 4 +- test/unit/obj/test_auditor.py | 286 +- test/unit/obj/test_diskfile.py | 38 +- test/unit/obj/test_expirer.py | 10 - test/unit/obj/test_replicator.py | 10 +- test/unit/obj/test_server.py | 271 +- test/unit/obj/test_ssync_sender.py | 4 +- test/unit/obj/test_updater.py | 7 +- test/unit/proxy/controllers/test_base.py | 681 ++-- test/unit/proxy/controllers/test_container.py | 2470 ++++++++------ test/unit/proxy/controllers/test_obj.py | 842 ++++- test/unit/proxy/test_server.py | 439 ++- tools/playbooks/common/install_dependencies.yaml | 26 +- tools/test-setup.sh | 12 - tox.ini | 4 +- 168 files changed, 17546 insertions(+), 7597 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index e9d54faaf..892fd5f79 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5 +5 @@ -eventlet>=0.25.0 # MIT +eventlet>=0.25.0,!=0.34.3 # MIT diff --git a/test-requirements.txt b/test-requirements.txt index 488080ae5..1e77045c9 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -6 +6 @@ -hacking>=2.0,<2.1.0 # Apache-2.0 +hacking>=2.0,<6.2.0 # Apache-2.0