We eagerly announce the release of: swift 2.30.0: OpenStack Object Storage This release is part of the zed release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.30.0 ^^^^^^ New Features ************ * Sharding improvements * The "swift-manage-shard-ranges" tool has a new mode to repair gaps in the namespace. * Metrics are now emitted for whether databases used for cleaving were created or already existed, allowing a better understanding of the reason for handoffs in the cluster. * Misplaced-record stats are now also emitted to statsd. Previously, these were only available in logs. * Logging improvements * The message template for proxy logging may now include a "{domain}" field for the client-provided "Host" header. * Added a "log_rsync_transfers" option to the object-replicator. Set it to false to disable logging rsync "send" lines; during large rebalances, such logging can overwhelm log aggregation while providing little useful information. * The formpost digest algorithm is now configurable via the new "allowed_digests" option, and support is added for both SHA-256 and SHA-512. Supported formpost digests are exposed to clients in "/info". Additionally, formpost signatures can now be base64 encoded. * Added metrics to the formpost and tempurl middlewares to monitor digest usage in signatures. * Improved compatibility with certain FIPS-mode-enabled systems. * Added a "ring_ip" option for various object services. This may be used to find own devices in the ring in a containerized environment where the "bind_ip" may not appear in the ring at all. * Account and container replicators can now be configured with a "handoff_delete" option, similar to object replicators and reconstructors. See the sample config for more information. * Developers using Swift's memcache client may now opt in to having a "MemcacheConnectionError" be raised when no connection succeeded using a new "raise_on_error" keyword argument to "get"/"set". * Device names are now included in new database IDs. This provides more context when examining incoming/outgoing sync tables or sharding CleaveContexts. Deprecation Notes ***************** * SHA-1 signatures are now deprecated for the formpost and tempurl middlewares. At some point in the future, SHA-1 will no longer be enabled by default; eventually, support for it will be removed entirely. Security Issues *************** * Constant-time string comparisons are now used when checking S3 API signatures. * Fixed a socket leak when clients try to delete a non-SLO as though it were a Static Large Object. Bug Fixes ********* * Sharding improvements * Misplaced tombstone records are now properly cleaved. * Fixed a bug where the sharder could fail to find a device to use for cleaving. * Databases marked deleted are now processed by the sharder. * More information is now synced to the fresh database when sharding. Previously, a database could lose the fact that it had been marked as deleted. * Shard ranges with no rows to cleave could previously be left in the CREATED state after cleaving. Now, they are advanced to CLEAVED. * S3 API improvements * Fixed cross-policy object copies. Previously, copied data would always be written using the source container's policy. Now, the destination container's policy will be used, avoiding availability issues and unnecessary container-reconciler work. * More headers are now copied from multi-part upload markers to their completed objects, including "Content-Encoding". * When running with "s3_acl" disabled, "bucket-owner-full-control" and "bucket-owner-read" canned ACLs will be translated to the same Swift ACLs as "private". * The S3 ACL and Delete Multiple APIs are now less case-sensitive. * Improved the error message when deleting a bucket that's ever had versioning enabled and still has versions in it. * "LastModified" timestamps in listings are now rounded up to whole seconds, like they are in responses from AWS. * Proxy logging for Complete Multipart Upload requests is now more consistent when requests have been retried. * Logging improvements * Signal handling is more consistently logged at notice level. Previously, signal handling would sometimes be logged at info or error levels. * The object-replicator now logs successful rsync transfers at debug instead of info. * Transaction IDs are now only included in daemon log lines in a request/response context. * The tempurl middleware has been updated to return a 503 if storing a token in memcache fails. Third party authentication middlewares are encouraged to also use the new "raise_on_error" keyword argument when storing ephemeral tokens in memcache. * Database replication connections are now closed following an error or timeout. This prevents a traceback in some cases when the replicator tries to reuse the connection. * "ENOENT" and "ENODATA" errors are better handled in the object replicator and auditor. * Improved object update throughput by shifting some shard range filtering from Python to SQL. * Include "Vary: Origin" header when CORS responses vary by origin. * The staticweb middleware now allows empty listings at the root of a container. Previously, this would result in a 404 response. * Ring builder output tables better display weights over 1000. * Various other minor bug fixes and improvements. Other Notes *********** * Pickle support has been removed from Swift's memcache client. Support had been deprecated since Swift 1.7.0. Changes in swift 2.29.1..2.30.0 ------------------------------- f6196b0a2 AUTHORS/CHANGELOG for 2.30.0 6fd523947 Fix misuse of assertTrue 9abee0e78 Fix docker image building dd99514e6 remove unicode prefix from code 9aa740f4f Imported Translations from Zanata d24678dd5 s3api: Be more consistent about CompleteMultipartUpload logging 3a7b89506 Imported Translations from Zanata 3a71df133 Stop using unicode literals in docs conf.py 1c577fed1 ring-builder: Better format large weights 52a4fe37a Various doc formatting cleanups 7e5c78423 Update "Getting Started" requirements 91cfb0d6d Imported Translations from Zanata 517738ac9 sharder/replicator: emit stats for DBs created or existing c4e00eb89 Sharder: Fall back to local device in get_shard_broker 59508de0c CI: Add nslookup_target to FIPS jobs e6ee37274 slo: Reduce overhead for 'Not an SLO manifest' responses 38271142e sharder: process deleted DBs 21fab529c sharder: emit misplaced stats to statsd 2d063cd61 formpost: deprecate sha1 signatures bc3625142 py310: Fix formatdate() call bf4edefce DB Replicator: Add handoff_delete option a5c1444fa Drain and close response in StaticLargeObject.get_slo_segments 25b6bd9f2 tempurl: Continue allowing sha1 by default 45e13ff4c Sharding: Sync container_stat table with fresh db 57f7145f7 sharder: always set state to CLEAVED after cleaving d7c08d8ea Make the object replicator and auditor more robust to ENODATA ac8f5550a sharder: fix probe tests skipping conditions a55016e57 Imported Translations from Zanata 475cdba65 Emit metrics for tempurl & formpost digest usage 6af444926 s3api: Better handle 503s in get_container_info, too 1831658b3 proxy-logging: Allow to add domain in log messages b9f1f4d60 Ensure clean starting state in test 91317ec14 Imported Translations from Zanata 367583c9f s3api: Make grentee types case insensitive 7a996a5c3 Fix s3api cross policy copy ef31baf3f formpost: Add support for sha256/512 signatures 68e5a0b1c tests: Fix cross-test interaction 24648528a doc: Comment out language option de13220c6 more tests for canned acls 12bc79bf0 Add ring_ip option to object services 5d9f1f009 s3api tests: allow AWS credential file loading 888142960 object-replicator: Remove some dead code c33b3d860 s3api: Add best-effort support for more canned ACLs 27db5213d CI: Run s3api test suite 5112cf712 Add Jianjian to authors. 1a5e6d5c1 Update AUTHORS 19855db95 pytest: explicitly set system logger to DEBUG 238dc0353 CI: constrain py36 deps b45b45fa7 manage-shard-ranges: add gap repair option d0cf743b6 ceph-tests: Remove known-failure 019c955e1 sharder: ensure that misplaced tombstone rows are moved d2b0c04d3 Add missing services to sample rsyslog.conf 75c5dbc29 trivial: add comment re sharder misplaced found stat 2f607cd31 Round s3api listing LastModified to integer resolution 52254bb5c Add ceil method to utils.Timestamp 99a4b9c7e AbstractRateLimiter: add option to burst on start-up 5227cb702 Refactor rate-limiting helper into a class 185b11e2f container-server: plumb includes down into _get_shard_range_rows 0b1cc8b0c More tests for rebalance_missing_suppression_count 94226bdd3 Don't give clients made up tokens ab612dd26 tests: Save ourselves 20s of sleep 05b2e894a Log signal handling at notice 7298038ed Ignore py36 deprecation warnings 9bed525bf memcached: Give callers the option to accept errors 7e6917681 replicator: Log rsync file transfers less 043e0163e Clarify that rsync_io_timeout is also used for contimeout 0708edecd Drop arm64 probe test job 11b9761cd Rip out pickle support in our memcached client 118cf2ba8 tempurl: Deprecate sha1 signatures 0bf5474bf ceph tests: Register output/ceph-s3-summary.log as a job output bb220f6f4 Doc: Update links in associated projects 5a272421d Swauth is retired 179fc43eb s3api: Improve error message when bucket is not empty 6142ce88c s3api: Use constant-time string comparisons in check_signature 1cee51d52 doc: also add reverse option to pagination doc 5c3bf6d26 replicator: Tolerate ENOENT when calling listdir f6f474e42 db: Close ReplConnection sockets on errors/timeouts a5a98d7e3 tests: Fix swiftclient/requests log level adjustment d496d03b7 api-ref: Document `reverse` param d29cbc399 CI: Run ceph and rolling upgrade tests under py3 fce7ad5f1 Ring: Change py2 only tests to py3 f92be1bdd Obj Auditor: Quarantine ENODATA 1c4acf2d8 s3api: Copy more headers from MPU marker to final object fd2dd1156 s3api: Make the 'Quiet' key value case insensitive ffb173f8a CI: Run CORS tests under py3 f83bfe1df Update master for stable/yoga 471a559a4 Stop partial()ing hashlib.new 8155e69b6 sharder: fix and expand CleavingContext docstrings bab7f9322 cors: Include `Vary: Origin` when using the request's Origin d94ab813f diskfile: Quarantine hashdirs on ENODATA 08da83c19 DB: Encode the device to the DB id d13eeabdb Clear logger txn_id after making requests 57e41685b trivial: Replace assertRegexpMatches with assertRegex 1e410347f trivial: Replace assertRaisesRegexp with assertRaisesRegex 10767e482 staticweb: Allow empty listings at the root of a container Diffstat (except docs and test files) ------------------------------------- .mailmap | 2 + .zuul.yaml | 76 +- AUTHORS | 7 +- CHANGELOG | 137 ++++ Dockerfile | 4 +- Dockerfile-py3 | 4 +- api-ref/source/parameters.yaml | 8 + api-ref/source/storage-account-services.inc | 1 + api-ref/source/storage-container-services.inc | 1 + .../pseudo-hierarchical-folders-directories.rst | 112 +-- .../api/use_the_content-disposition_metadata.rst | 20 +- etc/account-server.conf-sample | 13 +- etc/container-server.conf-sample | 13 +- etc/memcache.conf-sample | 10 - etc/object-server.conf-sample | 15 +- etc/proxy-server.conf-sample | 12 - etc/swift-rsyslog.conf-sample | 5 + py2-constraints.txt | 1 + py36-constraints.txt | 88 +++ .../notes/2_30_0_release-642778c3010848db.yaml | 167 +++++ releasenotes/source/conf.py | 6 +- releasenotes/source/index.rst | 2 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 598 +++++++++++++++- releasenotes/source/yoga.rst | 6 + swift/__init__.py | 4 + swift/account/backend.py | 3 +- swift/cli/manage_shard_ranges.py | 122 +++- swift/cli/ringbuilder.py | 8 +- swift/common/daemon.py | 2 +- swift/common/db.py | 6 +- swift/common/db_auditor.py | 7 +- swift/common/db_replicator.py | 35 +- swift/common/digest.py | 151 ++++ swift/common/internal_client.py | 5 +- swift/common/memcached.py | 55 +- swift/common/middleware/crypto/decrypter.py | 2 +- swift/common/middleware/crypto/keymaster.py | 8 +- swift/common/middleware/formpost.py | 45 +- swift/common/middleware/memcache.py | 15 - swift/common/middleware/proxy_logging.py | 8 + swift/common/middleware/s3api/acl_utils.py | 7 +- .../common/middleware/s3api/controllers/bucket.py | 12 +- .../middleware/s3api/controllers/multi_delete.py | 5 +- .../middleware/s3api/controllers/multi_upload.py | 13 +- swift/common/middleware/s3api/s3request.py | 13 +- swift/common/middleware/s3api/s3response.py | 6 + swift/common/middleware/s3api/schema/delete.rng | 2 +- swift/common/middleware/s3api/subresource.py | 11 +- swift/common/middleware/s3api/utils.py | 13 +- swift/common/middleware/slo.py | 14 +- swift/common/middleware/staticweb.py | 2 +- swift/common/middleware/tempauth.py | 42 +- swift/common/middleware/tempurl.py | 122 ++-- swift/common/ring/builder.py | 2 +- swift/common/storage_policy.py | 4 +- swift/common/utils.py | 246 ++++--- swift/common/wsgi.py | 9 +- swift/container/backend.py | 86 ++- swift/container/replicator.py | 47 +- swift/container/server.py | 5 +- swift/container/sharder.py | 510 +++++++------ swift/container/sync.py | 44 +- swift/container/updater.py | 8 +- swift/locale/de/LC_MESSAGES/swift.po | 764 +------------------- swift/locale/en_GB/LC_MESSAGES/swift.po | 796 +-------------------- swift/locale/es/LC_MESSAGES/swift.po | 682 +----------------- swift/locale/fr/LC_MESSAGES/swift.po | 589 +-------------- swift/locale/it/LC_MESSAGES/swift.po | 575 +-------------- swift/locale/ja/LC_MESSAGES/swift.po | 565 +-------------- swift/locale/ko_KR/LC_MESSAGES/swift.po | 558 +-------------- swift/locale/pt_BR/LC_MESSAGES/swift.po | 566 +-------------- swift/locale/ru/LC_MESSAGES/swift.po | 576 +-------------- swift/locale/tr_TR/LC_MESSAGES/swift.po | 533 +------------- swift/locale/zh_CN/LC_MESSAGES/swift.po | 535 +------------- swift/locale/zh_TW/LC_MESSAGES/swift.po | 539 +------------- swift/obj/auditor.py | 16 +- swift/obj/diskfile.py | 76 +- swift/obj/reconstructor.py | 4 +- swift/obj/replicator.py | 38 +- swift/obj/server.py | 5 +- swift/obj/ssync_receiver.py | 68 +- swift/obj/updater.py | 60 +- swift/proxy/controllers/base.py | 15 +- swift/proxy/controllers/container.py | 17 +- swift/proxy/controllers/info.py | 3 +- swift/proxy/controllers/obj.py | 6 +- test/__init__.py | 9 + test/debug_logger.py | 13 + test/functional/s3api/test_multi_upload.py | 87 +-- test/functional/s3api/test_object.py | 45 +- test/functional/s3api/test_presigned.py | 4 +- test/functional/test_object_versioning.py | 4 +- test/functional/test_symlink.py | 4 +- test/functional/test_tempurl.py | 37 +- test/probe/test_sharder.py | 498 ++++++++++++- test/s3api/__init__.py | 28 +- test/unit/__init__.py | 19 +- test/unit/account/test_backend.py | 37 +- test/unit/cli/test_ipv6_output.stub | 10 +- test/unit/cli/test_manage_shard_ranges.py | 392 +++++++++- test/unit/cli/test_ringbuilder.py | 2 +- .../common/middleware/crypto/test_keymaster.py | 2 +- test/unit/common/middleware/helpers.py | 16 +- .../unit/common/middleware/s3api/test_acl_utils.py | 51 +- test/unit/common/middleware/s3api/test_bucket.py | 83 ++- .../common/middleware/s3api/test_multi_delete.py | 40 +- .../common/middleware/s3api/test_multi_upload.py | 287 +++++--- test/unit/common/middleware/s3api/test_obj.py | 105 +-- test/unit/common/middleware/s3api/test_s3_acl.py | 11 + test/unit/common/middleware/s3api/test_s3api.py | 6 - .../unit/common/middleware/s3api/test_s3request.py | 16 +- test/unit/common/middleware/s3api/test_utils.py | 57 +- test/unit/common/middleware/test_formpost.py | 227 +++++- test/unit/common/middleware/test_memcache.py | 68 +- test/unit/common/middleware/test_proxy_logging.py | 6 +- test/unit/common/middleware/test_slo.py | 59 +- test/unit/common/middleware/test_staticweb.py | 20 + test/unit/common/middleware/test_tempauth.py | 9 + test/unit/common/middleware/test_tempurl.py | 167 +++-- test/unit/common/ring/test_builder.py | 26 +- test/unit/common/ring/test_ring.py | 135 ++-- test/unit/common/ring/test_utils.py | 21 +- test/unit/common/test_daemon.py | 12 +- test/unit/common/test_db_replicator.py | 71 +- test/unit/common/test_digest.py | 191 +++++ test/unit/common/test_internal_client.py | 35 +- test/unit/common/test_memcached.py | 92 ++- test/unit/common/test_utils.py | 176 ++++- test/unit/common/test_wsgi.py | 29 + test/unit/container/test_backend.py | 254 ++++++- test/unit/container/test_replicator.py | 78 ++ test/unit/container/test_sharder.py | 489 +++++++++++-- test/unit/obj/test_diskfile.py | 136 ++++ test/unit/obj/test_reconstructor.py | 54 +- test/unit/obj/test_replicator.py | 223 ++++++ test/unit/obj/test_ssync_receiver.py | 2 +- test/unit/obj/test_updater.py | 51 +- test/unit/proxy/controllers/test_container.py | 43 +- test/unit/proxy/controllers/test_info.py | 22 +- test/unit/proxy/controllers/test_obj.py | 25 + test/unit/proxy/test_server.py | 23 + tools/playbooks/common/restart_swift.yaml | 24 + tools/playbooks/cors/run.yaml | 2 +- tools/playbooks/multinode_setup/make_rings.yaml | 2 +- tools/playbooks/multinode_setup/pre.yaml | 1 - tools/playbooks/multinode_setup/run.yaml | 5 +- tox.ini | 19 + 182 files changed, 7341 insertions(+), 9491 deletions(-)