We are delighted to announce the release of: swift 2.29.0: OpenStack Object Storage This release is part of the yoga release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.29.0 ^^^^^^ New Features ************ * S3 API improvements * CORS preflights are now allowed for pre-signed URLs. * The "storage_domain" option now accepts a comma-separated list of storage domains. This allows multiple storage domains to configured for use with virtual-host style addressing. * Reduced the overhead of retrieving bucket and object ACLs. * Replication, reconstruction, and diskfile improvements * The reconstructor now uses the replication network to fetch fragments for reconstruction. * Added the ability to limit how many objects per handoff partition will be reverted in a reconstructor cycle using the new "max_objects_per_revert" option. This may be useful to reduce ssync timeouts and lock contention, ensuring that progress is made during rebalances. * Object updater improvements * Added the ability to ratelimit updates (approximately) per- container using the new "max_objects_per_container_per_second" option. This may be used to limit requests to already-overloaded containers while still making progress on updates to other containers. * Added timing stats by response code. * Updates are now sent over the replication network. * Memcache improvements * Added the ability to configure a chance to skip checking memcache when querying shard ranges. This allows some fraction of traffic to go to disk and refresh memcache before the key ages out. Recommended values for the new "container_updating_shard_ranges_skip_cache_pct" and "container_listing_shard_ranges_skip_cache_pct" options are in the range of 0.0 to 0.1. * Added stats for shard range cache hits, misses, and skips. * Added object-reconstructor stats to recon. * Added a new "swift.common.registry" module. This includes helper functions "register_sensitive_header" and "register_sensitive_param" which third party middleware authors may use to flag headers and query parameters for redaction when logging. For more information, see the documentation. * Added the ability to configure project-scope read-only roles for keystoneauth using the new "project_reader_roles" option. * The "cname_lookup" middleware now works with dnspython 2.0 and later. * The internal clients used by the container-reconciler, container- sharder, container-sync, and object-expirer daemons now use a more- descriptive "<daemon>-ic" log name, rather than "swift". If you previously configured the "log_name" option in "internal- client.conf", you must now use the "set log_name = <value>" syntax to configure it, even if no value is set in the "[DEFAULT]" section. This may be done prior to upgrading. * Removed translations from most logging. Deprecation Notes ***************** * The "StatsdClient.set_prefix" method is now deprecated and may be removed in a future release; by extension, so is the "LogAdapter.set_statsd_prefix" method. Middleware developers should use the "statsd_tail_prefix" argument to "get_logger" instead. Bug Fixes ********* * S3 API fixes * Fixed the types of configured values in "/info" response. * Fixed a server error when trying to copy objects with non-ASCII names. * Fixed a server error when uploading objects with very long names. A "KeyTooLongError" is now returned. * Fixed an error when multi-deleting MPUs when SLO async-deletes are enabled. * Fixed an error that allowed list-uploads and list-parts requests to return incomplete or out-of-order results. * Fixed several bugs when dealing with non-ASCII object names and multipart uploads. * Replication, reconstruction, and diskfile fixes * Ensure that non-durable data and .meta files are purged from handoffs after syncing. * Fixed tracebacks when there's a race to mark a file durable or delete it. * Improved cooperative multitasking during ssync. * Upon detecting a ring change, the reconstructor now only aborts the jobs for that ring and continues processing jobs for other rings. * Fixed a traceback when logging about a lock timeout in the replicator. * Fixed a security issue where tempurl and s3api signatures were logged in full. This allowed an attacker with access to log data to perform replay attacks, potentially accessing or overwriting cluster data. Now, such signatures are redacted in a manner similar to auth tokens; see the "reveal_sensitive_prefix" option in "proxy- server.conf". See CVE-2017-8761 for more information. * Fixed a race condition where swift would attempt to quarantine recently-deleted object updates. * Improved handling of timeouts and other errors when obtaining a connection to memcached. * The "swift-recon" tool now queries each object-server IP only once when reporting disk usage. Previously, each port in the ring would be queried; when using servers-per-port, this could dramatically overstate the disk capacity in the cluster. * Fixed a bug that allowed some statsd metrics to be annotated with the wrong backend layer. * Fixed a traceback in the account-server when there's no account database on disk to receive a container update. The account-server now correctly 404s. * The container-updater will quarantine container databases if all replicas for the account respond 404. * Fixed a proxy-server error when the read-only middleware tried to handle non-Swift paths (such as may be used by third-party middleware). * Some client behaviors that the proxy previously logged at warning have been lowered to info. * Various other minor bug fixes and improvements. Changes in swift 2.28.0..2.29.0 ------------------------------- 975d3dbcf AUTHORS/CHANGELOG for 2.29.0 c4762acaa Quiet more BadStatusLine tracebacks a9565893f Add docs for registry module 00bb0f8ba read-only: Only act on Swift paths 086aa5c4f CI: fix lower-constraints job f2c279bae Trim sensitive information in the logs (CVE-2017-8761) eda7d5fe3 Deprecate LogAdapter.set_statsd_prefix 7f2adb4e8 CI: Use xena on CentOS 8 Stream 589ac355f Move *_swift_info functions into a new registry module 363aa3319 Fix multipart upload listings 5f25e1cc7 s3api: Fix non-ascii MPUs ab52b2f77 tests: Clean up some dangling timeouts 9bc1c008a Get rid of pipeline_property 114440487 proxy-server: add stats for backend shard_listing requests a50726f27 Make NodeIter require a logger argument 6942b25cc Fix statsd prefix mutation in proxy controllers 0a1955d88 s3api: tighten register_info unit test 4edf0cde4 s3api: Fix types in /info 8c6ccb5fd proxy: Add a chance to skip memcache when looking for shard ranges 876a7de6a De-clutter container sync sample internal client conf 03be71c44 Add FIPS CI jobs 11d102216 s3api: Allow multiple storage domains 7be5cb0ab Error limit the correct node on ECFragGetter resume b8d7c3dcb Do not fetch content of container/object to retrieve S3 ACLs 035d91dce Modify log_name in internal clients' pipeline configs 1907594bd reconstructor: Abort just the changed policies 874a5865b tests: Improve FakeMemcache call tracking de8886298 Finer grained ratelimit for update f7101f379 tests: Unify FakeMemcaches 5079d8429 internal-client: pass global_conf to loadapp b1f03149f Attempt fix test timeouts 40e0f577a Add stats for shard range cache hits/misses 54fc8a7de Fix cname_lookup test ac92b3d8d CI: Add rolling upgrade job coming from stable/xena d6206e6fe CI: Move py37 and py38 jobs to experimental pipeline 546c9629e Move CI from CentOS 8 to CentOS 8 Stream 219a79a24 updater: Add timing stats eec595b12 updaters: use replication network 1b3879e0d reconstructor: include partially reverted handoffs in handoffs_remaining 8ee631cce reconstructor: restrict max objects per revert job 9acc44b15 Add some extra logging when EC decode fails 5979b6e15 account: Remove logging translations 7cfacbb25 container: Remove a bunch of logging translations 80d025233 object: Remove a bunch of logging translations 1942262db recon: only query each host once for disk usage 1eaf7474f Fix some imports for py310 346f518d6 Make arm jobs voting (but not the pipeline) ada9f0eeb reconstructor: purge meta files in pure handoffs 092d409c4 reconstructor: silence traceback when purging 88eb360f5 ssync sender: add context to missing_check error log e3069e6f7 reconstructor: remove non-durable files on handoffs 591adc845 Fix SSYNC update phase blocking bb487bb20 Fix SSYNC/missing_check blocking (sender) 3ee262d8b Make cmp_policy_info agree with the API behaviour 3d190dba5 Clean up devstack job definitions 439fbbdc7 Ensure close socket for memcached if got timeout 898a1d790 memcache: Prevent possible pool exhaustion 8ebd0354a Test for PartitionLockTimeout c4dca39a7 Make SAIO reconciler multiprocess 336599842 Fix log message interpolation f1c411c52 proxy: Remove a bunch of logging translations 7f5e34a40 sharidng: update doc to only mention auto_shard experimental d49dc9db5 Update master for stable/xena 4f84e77a6 Bump up timeout on swift-probetests-centos-8-arm64 job a5fbe6ca4 ec: Use replication network to get frags for reconstruction 1cde3938a cname_lookup: Work with dnspython 2.0+ 5b9a90b65 sharder: Make stats interval configurable 85d021127 Get rid of port to node assumptions and their modulo kludges c15818f1e s3api: fix the copy of non-ASCII objects 47749cd0e Bug: fix s3api multipart parts listings b4e532a46 func test improvements 9facfdc95 Clean up extra spaces dfd1bdc9e Add documentation for DELETE method for Swift Object Store API. 930ac1114 Fix docker image builds 7a105b5ef Add and pipe reconstructor stats through recon 927691098 Plumb allow_modify_pipeline through run_wsgi/run_server 2992f1417 container GET: return 503 if policy index mismatches 56510ab3c container-server: return objects of a given policy bcff1282b Band-aid and test the crash of the account server 619828483 Add a project scope read-only role to keystoneauth 61dd2ee44 s3api: Fix (async) multi-delete of MPUs 2a806296f func tests: plumb tolerate_missing through to listing_empty c9b0795b9 diskfile: Stop raising ENOENT when we race to mark durable 460dcf756 s3api: Allow CORS preflights for pre-signed URLs 66ab1da99 updater: Stop trying to quarantine missing asyncs ca0c34445 s3api: Return KeyTooLongError when upload name exceeds constraints a61d154fa proxy: Downgrade some client problems to info 0d44328c7 Quiet more deprecation warnings on py2 9f9360833 use a context manager to apply --no-auto-shard option Diffstat (except docs and test files) ------------------------------------- .zuul.yaml | 119 ++-- AUTHORS | 7 + CHANGELOG | 143 ++++- Dockerfile | 2 +- Dockerfile-py3 | 2 +- api-ref/source/storage-account-services.inc | 64 +++ .../1.conf} | 2 + etc/container-server.conf-sample | 3 + etc/internal-client.conf-sample | 4 +- etc/object-server.conf-sample | 24 + etc/proxy-server.conf-sample | 42 +- lower-constraints.txt | 2 +- py2-constraints.txt | 1 + .../notes/2_29_0_release-af71f7efd73109b0.yaml | 167 ++++++ releasenotes/source/index.rst | 2 + releasenotes/source/xena.rst | 6 + requirements.txt | 2 +- swift/__init__.py | 4 + swift/account/auditor.py | 5 +- swift/account/server.py | 10 +- swift/cli/container_deleter.py | 7 +- swift/cli/recon.py | 126 +++-- swift/common/internal_client.py | 25 +- swift/common/memcached.py | 32 +- swift/common/middleware/account_quotas.py | 2 +- swift/common/middleware/bulk.py | 4 +- swift/common/middleware/cname_lookup.py | 5 +- swift/common/middleware/container_quotas.py | 2 +- swift/common/middleware/container_sync.py | 3 +- swift/common/middleware/crossdomain.py | 2 +- swift/common/middleware/crypto/__init__.py | 3 +- swift/common/middleware/domain_remap.py | 4 +- swift/common/middleware/etag_quoter.py | 3 +- swift/common/middleware/formpost.py | 7 +- swift/common/middleware/keystoneauth.py | 22 +- swift/common/middleware/name_check.py | 3 +- swift/common/middleware/proxy_logging.py | 34 +- swift/common/middleware/ratelimit.py | 3 +- swift/common/middleware/read_only.py | 10 +- swift/common/middleware/recon.py | 9 + swift/common/middleware/s3api/acl_handlers.py | 3 + .../common/middleware/s3api/controllers/bucket.py | 4 +- .../middleware/s3api/controllers/multi_delete.py | 11 +- .../middleware/s3api/controllers/multi_upload.py | 98 ++-- swift/common/middleware/s3api/controllers/obj.py | 8 +- .../common/middleware/s3api/controllers/s3_acl.py | 2 +- .../middleware/s3api/controllers/versioning.py | 3 +- swift/common/middleware/s3api/s3api.py | 66 ++- swift/common/middleware/s3api/s3request.py | 24 +- swift/common/middleware/s3api/s3response.py | 7 +- swift/common/middleware/s3api/utils.py | 2 +- swift/common/middleware/slo.py | 7 +- swift/common/middleware/staticweb.py | 3 +- swift/common/middleware/symlink.py | 3 +- swift/common/middleware/tempauth.py | 12 +- swift/common/middleware/tempurl.py | 8 +- .../common/middleware/versioned_writes/__init__.py | 4 +- swift/common/registry.py | 154 ++++++ swift/common/swob.py | 6 +- swift/common/utils.py | 216 ++++---- swift/common/wsgi.py | 88 ++- swift/container/reconciler.py | 16 +- swift/container/server.py | 25 +- swift/container/sharder.py | 63 ++- swift/container/sync.py | 88 ++- swift/container/updater.py | 55 +- swift/obj/auditor.py | 31 +- swift/obj/diskfile.py | 119 ++-- swift/obj/expirer.py | 42 +- swift/obj/reconstructor.py | 151 +++-- swift/obj/replicator.py | 70 +-- swift/obj/server.py | 17 +- swift/obj/ssync_receiver.py | 4 + swift/obj/ssync_sender.py | 41 +- swift/obj/updater.py | 239 ++++++-- swift/proxy/controllers/account.py | 7 +- swift/proxy/controllers/base.py | 135 +++-- swift/proxy/controllers/container.py | 117 +++- swift/proxy/controllers/info.py | 4 +- swift/proxy/controllers/obj.py | 221 ++++---- swift/proxy/server.py | 56 +- test/__init__.py | 4 + test/cors/test-s3-obj.js | 25 + test/functional/s3api/__init__.py | 7 +- test/functional/s3api/s3_test_client.py | 23 +- test/functional/s3api/test_bucket.py | 12 +- test/functional/s3api/test_multi_delete.py | 4 +- test/functional/s3api/test_multi_upload.py | 27 +- test/functional/s3api/test_object.py | 15 +- test/functional/swift_test_client.py | 8 +- test/probe/common.py | 27 +- test/probe/test_container_failures.py | 30 +- test/probe/test_container_sync.py | 2 +- test/probe/test_object_failures.py | 2 +- test/probe/test_object_handoff.py | 33 +- test/probe/test_orphan_container.py | 150 +++++ test/probe/test_reconstructor_revert.py | 4 +- test/probe/test_sharder.py | 70 ++- test/unit/__init__.py | 58 +- test/unit/cli/test_container_deleter.py | 10 + test/unit/cli/test_recon.py | 83 ++- test/unit/common/middleware/crypto/test_crypto.py | 32 +- test/unit/common/middleware/helpers.py | 2 + test/unit/common/middleware/s3api/__init__.py | 2 +- test/unit/common/middleware/s3api/test_acl.py | 12 +- .../common/middleware/s3api/test_multi_delete.py | 23 +- .../common/middleware/s3api/test_multi_upload.py | 175 ++++-- test/unit/common/middleware/s3api/test_obj.py | 37 +- test/unit/common/middleware/s3api/test_s3api.py | 47 +- .../unit/common/middleware/s3api/test_s3request.py | 55 +- test/unit/common/middleware/s3api/test_utils.py | 4 +- test/unit/common/middleware/test_bulk.py | 8 +- test/unit/common/middleware/test_cname_lookup.py | 17 +- test/unit/common/middleware/test_container_sync.py | 3 +- test/unit/common/middleware/test_domain_remap.py | 10 +- test/unit/common/middleware/test_keystoneauth.py | 40 +- test/unit/common/middleware/test_name_check.py | 10 +- test/unit/common/middleware/test_proxy_logging.py | 297 +++++++--- test/unit/common/middleware/test_ratelimit.py | 52 +- test/unit/common/middleware/test_read_only.py | 54 +- test/unit/common/middleware/test_recon.py | 26 + test/unit/common/middleware/test_slo.py | 10 +- test/unit/common/middleware/test_symlink.py | 3 +- test/unit/common/middleware/test_tempauth.py | 68 +-- test/unit/common/middleware/test_tempurl.py | 38 +- .../common/middleware/test_versioned_writes.py | 10 +- test/unit/common/test_exceptions.py | 5 +- test/unit/common/test_internal_client.py | 140 ++++- test/unit/common/test_memcached.py | 35 ++ test/unit/common/test_registry.py | 296 ++++++++++ test/unit/common/test_utils.py | 401 +++++++------- test/unit/common/test_wsgi.py | 105 ++-- test/unit/container/test_reconciler.py | 126 ++++- test/unit/container/test_server.py | 124 +++++ test/unit/container/test_sharder.py | 39 +- test/unit/container/test_sync.py | 28 +- test/unit/container/test_updater.py | 54 +- test/unit/obj/test_diskfile.py | 130 ++++- test/unit/obj/test_expirer.py | 127 +++-- test/unit/obj/test_reconstructor.py | 451 ++++++++++++++- test/unit/obj/test_replicator.py | 15 + test/unit/obj/test_server.py | 4 + test/unit/obj/test_ssync.py | 2 +- test/unit/obj/test_ssync_sender.py | 181 +++++- test/unit/obj/test_updater.py | 437 +++++++++++++-- test/unit/proxy/controllers/test_base.py | 33 +- test/unit/proxy/controllers/test_container.py | 608 ++++++++++++++++++--- test/unit/proxy/controllers/test_info.py | 61 ++- test/unit/proxy/controllers/test_obj.py | 146 ++++- test/unit/proxy/test_server.py | 493 +++++++++++++---- tools/playbooks/multinode_setup/common_config.yaml | 2 +- tools/test-setup.sh | 2 +- 164 files changed, 7088 insertions(+), 2080 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 223fc617b..3b74c3a87 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8 +8 @@ netifaces>=0.8,!=0.10.0,!=0.10.1 -PasteDeploy>=1.3.3 +PasteDeploy>=2.0.0