We are stoked to announce the release of: swift 2.32.0: OpenStack Object Storage This release is part of the bobcat release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.32.0 ^^^^^^ New Features ************ * Python 3.11 is now supported. * Added the ability for reseller admins to set per-policy account quotas by posting metadata of the form "X-Account-Quota-Bytes- Policy-<policy name>". * Added a "keepalive_timeout" option to the proxy server to limit how long to wait for a client to initiate a request, separate from the general "client_timeout" option. Note that this requires eventlet 0.33.4 (currently unreleased) or later. * Added a "keep_cache_slo_manifest" option to the object server to better control whether SLO manifests are dropped from the page cache. * WSGI servers now accept a "--test-config" option that may be used to validate configuration changes before reloading/restarting the server. * Metrics improvements: * Metrics are now emitted for a variety of S3 error responses, in the form "s3api.<status_int>.<error_class>[.<reason>]" * Account and container info metrics now include the response status code when backend requests are made. * Added timing metrics to the container sharder for various operations. * A variety of performance improvements have been made for sharded container databases. * Various logging and metrics improvements when talking to memcache. * Improved formatting of meta and sysmeta for "swift-account-info" and "swift-container-info". Upgrade Notes ************* * Previously, under some circumstances, a non-standard config option such as "RECLAIM_AGE" might get parsed as "reclaim_age" for some processes but ignored by others. Now, all config parsing is case- sensitive; non-standard names will always be ignored. * The structure of cached shard ranges has changed, improving performance when listing or writing to sharded containers. Note that immediately after upgrade, the new structures will all be cache misses, which may lead to a thundering herd problem. To avoid this, upgrade just a few nodes first, let them service some fraction of traffic to populate the cache, then upgrade the rest of the cluster. Deprecation Notes ***************** * Removed the hard dependency on netifaces; it may still be used if the "getifaddrs" C function is not available. This fallback support may be removed in a future release. Bug Fixes ********* * Python 3 fixes: * Python 3 object servers can now read unencrypted non-ASCII metadata that was written under Python 2. * Ssync no longer corrupts unencrypted non-ASCII metadata during transfers. * Fixed an encoding issue when writing non-ASCII object names to sharded containers and shard range caching is not enabled. * Fixed an encoding issue when handling non-ASCII account names. * Fixed a "generator already executing" error on client disconnect. * Suppressed "RemoteDisconnected" tracebacks. * Fixed an issue that prevented proxy servers from emitting metrics and logs for backend requests made when getting account or container info. * Fixed ssync's handling of timestamp offsets. Previously, this could cause ssync to fail with a 409 Conflict, causing the transfer to fail and preventing handoffs from clearing. * Fixed an issue where an erasure-coded PUT could prevent other requests from being processed when network calls rarely or never blocked. * Fixed an issue when downloading an SLO manifest would hit a recoverable error and attempt to resume from another node. This would manifest as either a pyeclib decode error or an unexpected empty response. * The proxy server now applies error-limiting to the correct node when handling a recoverable node error. * Account, container, and object log fields are now correctly identified when returning "BadDigest" responses to S3 requests. * Reduced the backend load of making "?versions" requests to a container that has never had object versioning enabled. * The standard-library "logging" module is no longer monkey-patched when importing "swift.common.utils", making it easier to re-use swift code in other contexts. * Various other minor bug fixes and improvements. Changes in swift 2.31.1..2.32.0 ------------------------------- 191b627b8 Authors/ChangeLog for 2.32.0 f06e53695 Add rolling-upgrade job from Antelope 287fbadc1 tests: Pollute stderr less 1edf7df75 Partially revert "Pull libc-related functions out to a separate module" 336c64338 s3api: set swift.backend_path when returning 422 8f85e27c2 FakeSwift: use env['PATH_INFO'] to index uploaded objects 746daad33 FakeSwift: add tests 8bd21db25 FakeSwift: add docstring dab7192e1 tests: fix FakeSwift HEAD with query param ac524832e direct_client: support extra request params for direct_get_container. 9e065e2d2 fix swob HEAD handling for leak tracking e5d730dc5 proxy: Get rid of iter_nodes helper 6444ef9be ContainerBroker: clarify get_shard_ranges docstring 4a8879a36 proxy-server: fix AttributeError in exception handler 6251539dc proxy-server: de-duplicate implementation of is_good_source 629cd5751 proxy-server: clarify method naming in GET path fe09ef258 Save some effort when discovering used_source_etag f8a6dee62 proxy: encapsulate Getter resp, node and parts_iter 20dc88860 s3api: extra unit test assertion for statsd metric prefix 00bfc425c Add FakeStatsdClient to unit tests 0c9b545ea docs: Clean up proxy logging docs 27120b052 docs/SAIO: Make reconciler use the SAIO convention for logs dd664c47d doc/SAIO: Update the Fedora section cb468840b Add non-ascii meta values to ssync probe test 6e8d82c97 container-server: use LIMIT in get_own_shard_range() query bc300a516 proxy: add new metrics to account/container_info cache for skip/miss b42d90a34 py3: apply str_to_wsgi to includes in _get_shard_ranges c51e81f64 proxy: Bring back logging/metrics for get_*_info requests 8a5baf8e1 Imported Translations from Zanata 369a72c4c proxy: remove client_chunk_size and skip_bytes from GetOrHeadHandler 8e051c5b8 docs: Fix link to sphinx project page 46eeea11d Add a docstring for utils.Namespace 6d5301050 sharder: use ContainerBroker.has_other_shard_ranges() method bfbe8f909 Container-server: only check for shard range existence for DB state queries f7d6d5806 container-server: do [end_]marker filtering in SQL query bdbe8ce9f s3api: fix statsd prefix mutation 771d605d0 Add more unit tests for ranged GETs 365c0ef00 Encode header in latin-1 with wsgi_to_bytes cb1e584e6 Object-server: keep SLO manifest files in page cache. 1b7cf2947 Object-server: backfill unit test coverage for keep_cache_private 14a227f30 versioning: Only list versions container if it seems to exist e82536c64 Properly format meta and sysmeta in db info cli tools a82d9c58b Rename TestSource which is not a TestCase f9af0b70b s3api: emit metrics for error responses 20757612c proxy: Pop ignore-range backend header on resume 82b5335fd tests: Stop requiring <1ms test runtime 1c210d2e4 Change getting major:minor of blkdev 5ef169b92 Imported Translations from Zanata 68e48c868 proxy: error limit the correct node when resuming GETs e290d47c4 unify duplicate code in replication, EC GET paths 0e0db9003 sharder: Fix completed metric emission 483e17d5b py3: Quiet RemoteDisconnected tracebacks ca3f10770 tests: Fix replicator test for py311 f955e8104 CI: test under py311 0235db3d3 tests: Stop trying to mutate instantiated EntryPoints c5a3354fe Green GreenDBConnection.execute b46b735a3 Fix handling of non-ASCII accounts be0c481e4 Fix proxy traceback for GeneratorExit in py3 4c986d97b Use statsd_client more directly f150a5035 proxy: print logs when write new shard range cache into memcache. d2838ada4 proxy: Remove unused arg from set/clear_info_cache 7788c2c92 container: Stop pulling alternate stats when getting policy index 1f298714a docs: Format metrics in fixed-width font, not italics 307315bde docs: Move metric name/description tables out to separate page(s) 149b617c2 Fix GetOrHeadHandler unit tests to use public interface a25087e1c memcached: add method and time_spent to exception logs 49a5293cb memcached: handle failed incr+add+incr better effd4931d Use format_exc, not format_stack d3d503f90 Drop more translations 23fa18d30 Remove hard dependency on netifaces fd6a1ad65 Refactor for extract closure to method e29e2c3ae Move IP-address-related functions out to new module 776a9d9a1 Use the db for get_shard_usage 2deab27da s3api: Better logging for non-JSON when trying to do bucket listings f714f43c9 memcached: encapsulate common command parameters aa96cb3dc proxy: add periodic zero-time sleep during object PUT 4bc86ec30 Imported Translations from Zanata 66e6ee624 tests: Make dark data probe tests pass with sync_method = ssync cff092b68 docs: Update versions in Getting Started doc ca0dd8d70 docs: Remove references to out-dated install guides c4c9d5a40 move test_GET_pipeline to BaseTestObjectController 9adc3ce42 disable requests_mock pytest plugin 7b4642567 Memcached: emit memcache timing metrics when exceptions raised 647ee8390 Unit test for keepalive timeout 01a998a45 more explicit catch 9d98721e7 backend_ratelimit: Tighten blanket exception handling 744e9a94a Sharder: add timing metrics for individual steps and total time spent. d393fa482 testing xattr metadata with py3.8 780754096 Properly read py2 object metadata on py3 9fb860880 memcached: log user provided keys in exception error logging. 84b995f27 Don't monkey patch logging on import 8d23dd8ac Log (Watchdog's) Timeouts with duration ab03e057d Make all config parsing case-sensitive 9dfbac7c4 Error logs changed for ChunkWriteTimeout 88941ebe4 tests: Fix config numbers in test_versioning_with_metadata_replication 716ae48eb docs: Fix broken paste/pastedeploy links b8f0a0ed5 ring: Centralize device normalization 0518c6023 CI: Use [tox]requires to pin virtualenv 50c4ea032 ECFragGetter: assume policy.fragment_size is non-zero 0f95870c5 ECFragGetter: simplify iter_bytes_from_response_part 61b6ec263 Assert ChunkWriteTimeout errors are logged 74fc18d90 Extract some closures to methods 0a2bfe981 Use instance attr instead of non-local 105727bc8 tests: Fix test_cleanup_ondisk_files_commit_window bfc138627 CI: fix lower-constraints job e19a165a7 proxy controller: always pass x-backend-* headers to backend ed1f5193e docs: Clean up cross-domain doc formatting; call out CWE-942 469c38e9f wsgi: Add keepalive_timeout option 71d507f8e Proxy: restructure cached listing shard ranges 4e74e7f55 ssync: Round-trip offsets in meta/ctype Timestamps c95f8e6c0 tests for wsgi/daemon config parsing bbf9687b7 InternalClient: error if allow_modify_pipeline is True e5105ffa0 internal_client: Remove allow_modify_pipeline option 12bfcb5cd tests: Fix PriorityQueue import c78a5962b Pull libc-related functions out to a separate module f264b53a2 ring-builder: Abort cleanly on ^C, ^D 85d68ee49 Test nested (Metrics)PrefixLoggerAdapters 0a4e41701 Add cap_length helper 0a5f0253b Add --test-config option to WSGI servers c21256d87 Pull timestamp-related functions out to a separate module 91db804e6 Start common.utils refactor 0dff2cae9 Patch policies for more per-policy quota tests 6885b3367 sharder: fix host logged when send_shard_ranges fails e15d6880e sharder: fix 'Cleaving not completed' warning log 5032b42b1 Update url 56a9f72ec Forward-port stable-release CHANGELOG entries cbba65ac9 quotas: Add account-level per-policy quotas 29414ab14 Allow internal container POSTs to not update put_timestamp 9a1bfb897 proxy: add tests for listing with shard gaps acf31a61d Rename ShardRange*Bound to Namespace*Bound ea6c48321 Update master for stable/2023.1 052bcadb2 tests: Skip s3api functional tests when no s3api user configured 78f13be75 tests: Let func tests run with test users 1 and 2 but not 3 479a01016 Drop bandit check B309 6ff90ea73 Proxy: restructure cached updating shard ranges 2fe18b24c ssync: fix decoding of ts_meta when ts_data has offset Diffstat (except docs and test files) ------------------------------------- .zuul.yaml | 30 +- AUTHORS | 4 + CHANGELOG | 272 ++ README.rst | 2 +- bandit.yaml | 3 +- bin/swift-account-server | 2 +- bin/swift-container-server | 2 +- bin/swift-drive-audit | 59 +- bin/swift-object-server | 2 +- bin/swift-proxy-server | 2 +- etc/internal-client.conf-sample | 1 + etc/object-server.conf-sample | 5 + etc/proxy-server.conf-sample | 13 +- .../notes/2_32_0_release-39c8fb77a0a3e72d.yaml | 122 + releasenotes/source/2023.1.rst | 6 + releasenotes/source/index.rst | 14 +- .../locale/en_GB/LC_MESSAGES/releasenotes.po | 482 ++- requirements.txt | 1 - swift/cli/info.py | 11 +- swift/cli/ringbuilder.py | 6 +- swift/common/container_sync_realms.py | 13 +- swift/common/daemon.py | 10 +- swift/common/db.py | 29 +- swift/common/db_replicator.py | 39 +- swift/common/direct_client.py | 77 +- swift/common/internal_client.py | 27 +- swift/common/manager.py | 87 +- swift/common/memcached.py | 242 +- swift/common/middleware/__init__.py | 4 + swift/common/middleware/account_quotas.py | 138 +- swift/common/middleware/backend_ratelimit.py | 10 +- swift/common/middleware/crossdomain.py | 29 +- swift/common/middleware/proxy_logging.py | 21 +- .../common/middleware/s3api/controllers/bucket.py | 10 +- swift/common/middleware/s3api/s3api.py | 13 +- swift/common/middleware/s3api/s3request.py | 56 +- swift/common/middleware/s3api/s3response.py | 11 +- swift/common/middleware/tempauth.py | 19 +- .../common/middleware/versioned_writes/__init__.py | 3 - .../versioned_writes/object_versioning.py | 27 +- swift/common/request_helpers.py | 3 +- swift/common/ring/ring.py | 41 +- swift/common/swob.py | 3 +- swift/common/{utils.py => utils/__init__.py} | 1756 ++++------- swift/common/utils/ipaddrs.py | 256 ++ swift/common/utils/libc.py | 345 +++ swift/common/utils/timestamp.py | 399 +++ swift/common/wsgi.py | 109 +- swift/container/backend.py | 144 +- swift/container/server.py | 13 +- swift/container/sharder.py | 49 +- swift/locale/de/LC_MESSAGES/swift.po | 321 +- swift/locale/en_GB/LC_MESSAGES/swift.po | 348 +-- swift/locale/es/LC_MESSAGES/swift.po | 280 +- swift/locale/fr/LC_MESSAGES/swift.po | 210 +- swift/locale/it/LC_MESSAGES/swift.po | 208 +- swift/locale/ja/LC_MESSAGES/swift.po | 280 +- swift/locale/ko_KR/LC_MESSAGES/swift.po | 306 +- swift/locale/pt_BR/LC_MESSAGES/swift.po | 205 +- swift/locale/ru/LC_MESSAGES/swift.po | 204 +- swift/locale/tr_TR/LC_MESSAGES/swift.po | 193 +- swift/locale/zh_CN/LC_MESSAGES/swift.po | 193 +- swift/locale/zh_TW/LC_MESSAGES/swift.po | 194 +- swift/obj/diskfile.py | 25 +- swift/obj/server.py | 26 +- swift/obj/ssync_receiver.py | 28 +- swift/obj/ssync_sender.py | 20 +- swift/proxy/controllers/account.py | 14 +- swift/proxy/controllers/base.py | 740 ++--- swift/proxy/controllers/container.py | 165 +- swift/proxy/controllers/obj.py | 644 ++-- swift/proxy/server.py | 11 +- test/debug_logger.py | 135 +- test/functional/s3api/__init__.py | 4 + test/functional/test_dlo.py | 4 +- test/functional/test_domain_remap.py | 14 + test/functional/test_object_versioning.py | 3 +- test/functional/test_slo.py | 9 +- test/functional/test_versioned_writes.py | 81 +- test/probe/test_container_failures.py | 73 +- test/probe/test_dark_data.py | 3 +- test/probe/test_object_versioning.py | 99 +- test/probe/test_reconstructor_revert.py | 6 +- test/unit/__init__.py | 79 +- test/unit/account/test_auditor.py | 5 +- test/unit/account/test_backend.py | 28 +- test/unit/account/test_reaper.py | 9 +- test/unit/account/test_server.py | 2 +- test/unit/cli/test_info.py | 19 +- test/unit/common/middleware/helpers.py | 129 +- test/unit/common/middleware/s3api/__init__.py | 4 +- test/unit/common/middleware/s3api/test_bucket.py | 19 + .../common/middleware/s3api/test_multi_upload.py | 29 + test/unit/common/middleware/s3api/test_obj.py | 4 + test/unit/common/middleware/s3api/test_s3api.py | 234 +- test/unit/common/middleware/test_account_quotas.py | 93 + .../common/middleware/test_backend_ratelimit.py | 6 +- test/unit/common/middleware/test_dlo.py | 24 + test/unit/common/middleware/test_formpost.py | 2 +- test/unit/common/middleware/test_helpers.py | 450 +++ .../common/middleware/test_object_versioning.py | 133 +- test/unit/common/middleware/test_proxy_logging.py | 16 +- test/unit/common/middleware/test_slo.py | 47 +- test/unit/common/middleware/test_symlink.py | 13 +- test/unit/common/middleware/test_tempauth.py | 11 +- test/unit/common/middleware/test_tempurl.py | 2 +- test/unit/common/ring/test_ring.py | 16 +- test/unit/common/test_daemon.py | 126 +- test/unit/common/test_db_replicator.py | 10 +- test/unit/common/test_direct_client.py | 132 +- test/unit/common/test_internal_client.py | 82 + test/unit/common/test_memcached.py | 459 ++- test/unit/common/test_swob.py | 24 +- test/unit/common/test_utils.py | 3178 ++++++++------------ test/unit/common/test_wsgi.py | 261 +- test/unit/common/utils/__init__.py | 0 test/unit/common/utils/test_ipaddrs.py | 216 ++ test/unit/common/utils/test_libc.py | 167 + test/unit/common/utils/test_timestamp.py | 882 ++++++ test/unit/container/test_backend.py | 282 +- test/unit/container/test_replicator.py | 27 +- test/unit/container/test_server.py | 117 +- test/unit/container/test_sharder.py | 425 ++- test/unit/helpers.py | 1 + test/unit/obj/test_diskfile.py | 288 +- test/unit/obj/test_reconstructor.py | 14 +- test/unit/obj/test_replicator.py | 20 +- test/unit/obj/test_server.py | 371 ++- test/unit/obj/test_ssync.py | 32 +- test/unit/obj/test_ssync_receiver.py | 50 +- test/unit/obj/test_ssync_sender.py | 90 +- test/unit/obj/test_updater.py | 119 +- test/unit/proxy/controllers/test_base.py | 483 ++- test/unit/proxy/controllers/test_container.py | 328 +- test/unit/proxy/controllers/test_obj.py | 328 +- test/unit/proxy/test_server.py | 433 ++- tox.ini | 7 +- 177 files changed, 12701 insertions(+), 8746 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 3b74c3a87..e9d54faaf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7 +6,0 @@ greenlet>=0.3.2 -netifaces>=0.8,!=0.10.0,!=0.10.1