We are satisfied to announce the release of: swift 2.20.0: OpenStack Object Storage This release is part of the stein release series. Download the package from: https://tarballs.openstack.org/swift/ For more details, please see below. 2.20.0 ^^^^^^ New Features * S3 API compatibility updates * Swift can now cache the S3 secret from Keystone to use for subsequent requests. This functionality is disabled by default but can be enabled by setting the "secret_cache_duration" in the "[filter:s3token]" section of the proxy server config to a number greater than 0. * s3api now mimics the AWS S3 behavior of periodically sending whitespace characters on a Complete Multipart Upload request to keep the connection from timing out. Note that since a request could fail after the initial 200 OK response has been sent, it is important to check the response body to determine if the request succeeded. * s3api now properly handles "x-amz-metadata-directive" headers on COPY operations. * s3api now uses concurrency (default 2) to handle multi-delete requests. This allows multi-delete requests to be processed much more quickly. * s3api now mimics some forms of AWS server-side encryption based on whether Swift's at-rest encryption functionality is enabled. Note that S3 API users are now able to know more about how the cluster is configured than they were previously, ie knowledge of encryption at-rest functionality being enabled or not. * s3api responses now include a '-' in multipart ETags. For new multipart-uploads via the S3 API, the ETag that is stored will be calculated in the same way that AWS uses. This ETag will be used in GET/HEAD responses, bucket listings, and conditional requests via the S3 API. Accessing the same object via the Swift API will use the SLO Etag; however, in JSON container listings the multipart upload etag will be exposed in a new "s3_etag" key. Previously, some S3 clients would complain about download corruption when the ETag did not have a '-'. * S3 ETag for SLOs now include a '-'. Ordinary objects in S3 use the MD5 of the object as the ETag, just like Swift. Multipart Uploads follow a different format, notably including a dash followed by the number of segments. To that end (and for S3 API requests *only*), SLO responses via the S3 API have a literal '-N' added on the end of the ETag. * The default location is now set to "us-east-1". This is more likely to be the default region that a client will try when using v4 signatures. Deployers with clusters that relied on the old implicit default location of "US" should explicitly set "location = US" in the "[filter:s3api]" section of proxy-server.conf before upgrading. * Add basic support for ?versions bucket listings. We still do not have support for toggling S3 bucket versioning, but we can at least support getting the latest versions of all objects. * Fixed an issue with SSYNC requests to ensure that only one request can be running on a partition at a time. * Data encryption updates * The "kmip_keymaster" middleware can now be configured directly in the proxy-server config file. The existing behavior of using an external config file is still supported. * Multiple keymaster middlewares are now supported. This allows migration from one key provider to another. Note that "secret_id" values must remain unique across all keymasters in a given pipeline. If they are not unique, the right- most keymaster will take precedence. When looking for the active root secret, only the right-most keymaster is used. * Prevent PyKMIP's kmip_protocol logger from logging at DEBUG. Previously, some versions of PyKMIP would include all wire data when the root logger was configured to log at DEBUG; this could expose key material in logs. Only the "kmip_keymaster" was affected. * Fixed an issue where a failed drive could prevent the container sharder from making progress. * Storage policy definitions in swift.conf can now define the diskfile to use to access objects. See the included swift.conf- sample file for a description of usage. * The EC reconstructor will now attempt to remove empty directories immediately, while the inodes are still cached, rather than waiting until the next run. * Added a "keep_idle" config option to configure KEEPIDLE time for TCP sockets. The default value is the old constant of 600. * Add "databases_per_second" to the account-replicator, container- replicator, and container-sharder. This prevents them from using a full CPU core when they are not IO limited. * Allow direct_client users to overwrite the "X-Timestamp" header. * Various other minor bug fixes and improvements. Changes in swift 2.19.0..2.20.0 ------------------------------- fbad538 authors/changelog for 2.20.0 release defae1d Fix rolling upgrade job f98d5d1 Py3: Add working tests to tox.ini 7458f23 Python3: fix test_xprofile.py b0aea93 s3 secret caching bce2ac6 Fix container sharding doc fba94b2 fix a typo in docstring f765f4b Py3: Use BytesIO when we want bytes 00be3f5 s3api: Allow concurrent multi-deletes 8e95a93 s3api: Allow some forms of server-side-encryption b7e2278 Support long-running multipart uploads 3379235 api-ref: Clean up container and object response code formatting 9ab276d Change openstack-dev to openstack-discuss c26d67e fixed _check_node() in the container sharder a883ada Py3: Replace map() with a list comprehension 014d46f Fix SSYNC concurrency on partition 333ae30 Fix in-process func tests to work with newer PasteDeploy 010cdc2 Py3: Fix test_memcache 59d66e3 Py3: fix a simple bytes vs str issue 3a51771 Py3: Use dict.items instead of dict.iteritems d75d15b Py3: We cannot use len(filter(...)) 21a3423 Py3: Fix a relative import c3cbe69 Py3: Use collections.MutableMapping rather than UserDict.DictMixin 6c8b4c8 Py3: Port test/unit/test_ringcomposer.py 9017db4 Only print "SKIPPING FUNCTIONAL TESTS ..." messages once 34aef9d Add chaging secret key for external KMS section fda3052 Multi-key KMS keymaster 4809884 Use eventlet.patcher.original to get Python select module in get_hub 42b4cdc s3api: Include '-' in S3 ETags of normal SLOs 582f058 py3: encryption follow-up 37b8146 py3: port encryption 3465d63 Add missing ws seperator between words 370f7d7 Handle non-ASCII characters in S3 v2 listings. 692a034 s3api: Change default location to us-east-1 a85dc0a update docs link address c1c65a7 Only url-quote Keys when encoding-type=url 662fd81 Fix typo c5e33b0 Fix typo a2fb335 s3api: Add basic support for ?versions bucket listings 34e48d5 Remove setup.py check from pep8 job bc4494f py3: port account/container replicators c112203 py3: Monkey-patch json.loads to accept bytes on py35 7a21272 Update min tox version to 2.3.2 887ba87 Add "historical document" note to ring background docs 3420921 Clean up HASH_PATH_* patching 11e81cf added note about double url quoting 441df4f Use correct headers in reconstructor requests e8a7729 Fix usage of multiple accounts in functional tests 06cf5d2 Add databases_per_second to db daemons 2d1c438 SSYNC: stop sharing global available_map/send_map 6b94cf2 SSYNC: Stop sharing a global response e4ad56a SSYNC: Stop sharing a global connection 46c6fab SSYNC: Remove useless self.failures in sender d306345 Remove empty directories after a revert job 1663782 Fix up the test for .ismount 654187e py3: adapt common/db_replicator.py b5a9172 Require gettext for all non-SUSE distros 0a564d8 Check for .ismount stubs with symlinks, too 3bf9f82 Reduce code duplication in SSYNC sender 781da07 Stop accomodating 5+ year old object-server code in unit tests 6676d74 py3: everything account except the real things b39d2ef Allow multiple keymasters 00f7732 Simplify the decryption of container listings 0d77486 Unify handle_get/handle_head in decrypter 28a582f py3: adapt account/backend.py and utils.py 83a7ce8 Python 3 compatibility: fix xrange/range issues b90c792 Follow up s3api ensures any string for secret as bytes string 45c840a s3api: Ensure secret is utf8 in check_signature dd49f17 Add coverage reports to py27/35/36 unit test jobs ce51893 Tighten status code assertions for object POST 78c9fd9 Change PUT bucket conflict error 4ca605c Move base64-decoding/unpickling into DatabaseBroker 6e7ca26 DB follow-up 3770f3f internal_client: Require that request_tries be positive e3a5b63 py3: Adapt db.py 5558b92 Properly handle custom metadata upon an object COPY operation 12b6d46 Prevent kmip_protocol logger from logging at DEBUG 27d8523 Give better errors for malformed credentials c90d34b py3: port tempauth c8e3b99 Enabling direct_client users to overwrite X-Timestamp 9686680 Tolerate missing containers when trying to clean up 93fb77e Ignore ENOENT and ENOTEMPTY errors in delete_partition 8c7ca01 Unquote URL before using splited parts. 36a3fee update 1space url in associated projects ae49663 py3: port internal_client ddd5cc1 py3: port direct_client b7fda3b Allow kmip_keymaster to be configured in proxy-server.conf ecebea1 s3api: Increase max body size for Delete Multiple Objects requests 050f879 Use latest eventlet in probe tests 70d31e0 set up a lower constraints job that uses an XFS tmpdir 62f8e64 Use templates for cover and lower-constraints 1b0172f py3: Port listing_formats middleware 78344bf py3: get proxy-server willing and able to respond to some API requests 904e7c9 Add more doc and test for cors_expose_headers option 5d70396 add multinode functional test to swift 0c97888 fixed a manpage and added the manpage syntax checker to pep8 jobs 2ef21ac py3: port request_helpers a7dd5f6 Replace Chinese quotes with English quotes b4567e6 py3: include test_base_storage_server cfeb32c Adding keep_idle config value to socket dbacdcf Add punch_hole utility function 1142202 py3: port list_endpoints 5a8cfd6 Add another user for s3api func tests 80001aa Add ceph-s3 test non-voting job 123f6f5 Allow EC to ignore invalid request ETag cbfa585 Refactor obj.server.ObjectController.PUT 52ecbf9 Add a chunks_finished to BaseDiskFileWriter 84b85f0 s3api: Include '-' in multipart ETags ebcc95a added some bootstrapping info to the top of the SAIO doc ce257b3 DiskFile(Writer) refactor cleanups a42c1da DiskFile(Writer) refactor cleanup for unused use_linkat 33c7650 Add writer method to DiskFile 8bc1434 Import quote directly from swift.common.utils 7914826 Let error messages to be normal strings again 560db71 Drop neutron jobs from our gate 2ae00de Add py36 test job 77734bc Disable the docs badge saying the docs aren't accurate b4be37c Remove post_as_copy deprecated warning from copy middleware b478a07 Add IPv6 DSVM tests 8cab628 remove legacy-swift-dsvm-functional jobs 1f29508 Allow for backwards compatibility with swift3. 125bb87 Remove install-guide-jobs d562659 Tighten up pep8 irrelevant-files 6f00d42 bulk: Use make_subrequest to make subrequests 85f4f94 Factor out a proper BaseKeyMaster class 1d65219 Move legacy-swift-dsvm-functional job in-tree 1e800fc switch documentation job to new PTI 90e260e import zuul job settings from project-config 9167e95 Imported Translations from Zanata df954e2 py3: port bufferedhttp (hopefully) 673fda7 Configure diskfile per storage policy e547448 Fix warning while building docs 04d3d68 minor updates to changelog and release notes ae9536e Update reno for stable/rocky dd313b8 Reference legacy-tempest-dsvm-neutron-full and neutron-grenade in-tree 2ddc979 Add irrelevant-files lists for in-tree check/gate jobs 6994a2e ratelimit: ignore requests with invalid API versions b2fbc74 Clean up comment about 404s leaking out during COPYs e9d0a7a Replacing six.iter/dict.iter usages 9295ba8 Add validation for sorting_method values 51622fd Pass empty etag override values through encrypter Diffstat (except docs and test files) ------------------------------------- .mailmap | 1 + .zuul.yaml | 281 ++++++- AUTHORS | 13 +- CHANGELOG | 108 ++- CONTRIBUTING.rst | 2 +- api-ref/source/storage-container-services.inc | 15 +- api-ref/source/storage-object-services.inc | 17 +- bindep.txt | 4 +- etc/account-server.conf-sample | 4 + etc/container-server.conf-sample | 7 + etc/object-server.conf-sample | 1 + etc/proxy-server.conf-sample | 19 +- etc/swift.conf-sample | 7 + .../notes/2_10_0_release-666a76f4975657a5.yaml | 6 +- .../notes/2_19_0_release-3e6ee3e6a1fcc6bb.yaml | 6 +- .../notes/2_20_0_release-7b090a5f4bd916e4.yaml | 116 +++ releasenotes/source/index.rst | 2 + .../locale/ko_KR/LC_MESSAGES/releasenotes.po | 75 ++ releasenotes/source/rocky.rst | 6 + setup.cfg | 5 +- swift/__init__.py | 35 + swift/account/backend.py | 19 +- swift/account/reaper.py | 13 +- swift/account/utils.py | 2 +- swift/cli/manage_shard_ranges.py | 2 +- swift/common/bufferedhttp.py | 56 +- swift/common/constraints.py | 38 +- swift/common/db.py | 34 +- swift/common/db_replicator.py | 14 +- swift/common/direct_client.py | 55 +- swift/common/internal_client.py | 28 +- swift/common/memcached.py | 4 +- swift/common/middleware/bulk.py | 52 +- swift/common/middleware/copy.py | 20 +- swift/common/middleware/crypto/crypto_utils.py | 8 +- swift/common/middleware/crypto/decrypter.py | 123 ++-- swift/common/middleware/crypto/encrypter.py | 17 +- swift/common/middleware/crypto/keymaster.py | 216 +++--- swift/common/middleware/crypto/kmip_keymaster.py | 19 +- swift/common/middleware/crypto/kms_keymaster.py | 56 +- swift/common/middleware/listing_formats.py | 30 +- swift/common/middleware/ratelimit.py | 3 + swift/common/middleware/s3api/acl_handlers.py | 2 +- .../common/middleware/s3api/controllers/bucket.py | 125 +++- .../middleware/s3api/controllers/location.py | 2 +- .../middleware/s3api/controllers/multi_delete.py | 50 +- .../middleware/s3api/controllers/multi_upload.py | 173 +++-- swift/common/middleware/s3api/controllers/obj.py | 21 +- swift/common/middleware/s3api/etree.py | 16 +- swift/common/middleware/s3api/s3api.py | 92 ++- swift/common/middleware/s3api/s3request.py | 160 +++- swift/common/middleware/s3api/s3response.py | 47 +- swift/common/middleware/s3api/s3token.py | 182 +++-- swift/common/middleware/symlink.py | 4 +- swift/common/middleware/tempauth.py | 28 +- swift/common/middleware/x_profile/html_viewer.py | 10 +- swift/common/middleware/xprofile.py | 16 +- swift/common/request_helpers.py | 6 +- swift/common/ring/ring.py | 2 +- swift/common/storage_policy.py | 39 +- swift/common/swob.py | 8 +- swift/common/utils.py | 297 ++++++-- swift/common/wsgi.py | 10 +- swift/container/backend.py | 22 +- swift/container/reconciler.py | 8 +- swift/container/replicator.py | 5 +- swift/container/server.py | 9 +- swift/container/sharder.py | 21 +- swift/locale/en_GB/LC_MESSAGES/swift.po | 10 +- swift/locale/es/LC_MESSAGES/swift.po | 6 +- swift/locale/ja/LC_MESSAGES/swift.po | 6 +- swift/obj/diskfile.py | 234 +++--- swift/obj/mem_diskfile.py | 43 +- swift/obj/reconstructor.py | 18 +- swift/obj/replicator.py | 7 +- swift/obj/server.py | 393 ++++++---- swift/obj/ssync_receiver.py | 4 +- swift/obj/ssync_sender.py | 292 ++++---- swift/proxy/controllers/base.py | 18 +- swift/proxy/controllers/container.py | 2 +- swift/proxy/controllers/obj.py | 80 +- swift/proxy/server.py | 22 +- test/functional/__init__.py | 80 +- test/functional/s3api/s3_test_client.py | 7 +- test/functional/s3api/test_acl.py | 19 +- test/functional/s3api/test_bucket.py | 215 +++--- test/functional/s3api/test_multi_delete.py | 21 +- test/functional/s3api/test_multi_upload.py | 85 ++- test/functional/s3api/test_object.py | 29 + test/functional/swift_test_client.py | 19 +- test/functional/test_access_control.py | 4 +- test/functional/test_slo.py | 55 +- test/functional/test_symlink.py | 4 + test/functional/test_tempurl.py | 41 +- test/functional/test_versioned_writes.py | 112 +-- test/probe/test_signals.py | 18 +- test/sample.conf | 8 +- test/unit/__init__.py | 18 +- test/unit/account/test_backend.py | 37 +- test/unit/account/test_reaper.py | 6 +- test/unit/cli/test_relinker.py | 21 +- test/unit/cli/test_ringcomposer.py | 2 +- .../common/middleware/crypto/crypto_helpers.py | 14 +- .../common/middleware/crypto/test_crypto_utils.py | 199 ++--- .../common/middleware/crypto/test_decrypter.py | 272 +++---- .../common/middleware/crypto/test_encrypter.py | 201 +++-- .../common/middleware/crypto/test_encryption.py | 59 +- .../common/middleware/crypto/test_keymaster.py | 144 +++- .../middleware/crypto/test_kmip_keymaster.py | 74 +- .../common/middleware/crypto/test_kms_keymaster.py | 73 +- test/unit/common/middleware/helpers.py | 8 +- test/unit/common/middleware/s3api/__init__.py | 15 +- test/unit/common/middleware/s3api/helpers.py | 15 +- test/unit/common/middleware/s3api/test_acl.py | 4 +- test/unit/common/middleware/s3api/test_bucket.py | 209 ++++-- .../common/middleware/s3api/test_multi_delete.py | 89 ++- .../common/middleware/s3api/test_multi_upload.py | 241 +++++- test/unit/common/middleware/s3api/test_obj.py | 114 ++- test/unit/common/middleware/s3api/test_s3api.py | 285 ++++++-- .../unit/common/middleware/s3api/test_s3request.py | 83 ++- .../common/middleware/s3api/test_s3response.py | 7 +- test/unit/common/middleware/s3api/test_s3token.py | 136 ++++ test/unit/common/middleware/s3api/test_service.py | 5 +- test/unit/common/middleware/test_copy.py | 49 -- test/unit/common/middleware/test_list_endpoints.py | 31 +- .../unit/common/middleware/test_listing_formats.py | 84 +-- test/unit/common/middleware/test_memcache.py | 2 +- test/unit/common/middleware/test_ratelimit.py | 48 +- test/unit/common/middleware/test_tempauth.py | 97 ++- test/unit/common/middleware/test_xprofile.py | 6 +- test/unit/common/test_bufferedhttp.py | 37 +- test/unit/common/test_daemon.py | 4 +- test/unit/common/test_db.py | 58 +- test/unit/common/test_db_replicator.py | 8 +- test/unit/common/test_direct_client.py | 264 +++---- test/unit/common/test_internal_client.py | 206 +++--- test/unit/common/test_request_helpers.py | 57 +- test/unit/common/test_storage_policy.py | 86 ++- test/unit/common/test_utils.py | 809 ++++++++++++++------- test/unit/common/test_wsgi.py | 35 +- test/unit/container/test_replicator.py | 34 +- test/unit/container/test_sharder.py | 135 +++- test/unit/container/test_sync.py | 4 +- test/unit/container/test_updater.py | 4 +- test/unit/helpers.py | 42 +- test/unit/obj/test_auditor.py | 2 +- test/unit/obj/test_diskfile.py | 215 ++++-- test/unit/obj/test_reconstructor.py | 73 +- test/unit/obj/test_replicator.py | 4 +- test/unit/obj/test_server.py | 25 +- test/unit/obj/test_ssync.py | 13 +- test/unit/obj/test_ssync_receiver.py | 59 +- test/unit/obj/test_ssync_sender.py | 487 ++++++------- test/unit/obj/test_updater.py | 4 +- test/unit/proxy/controllers/test_info.py | 10 +- test/unit/proxy/controllers/test_obj.py | 11 + test/unit/proxy/test_server.py | 88 ++- tools/playbooks/common/cover-post.yaml | 3 + tools/playbooks/common/install_dependencies.yaml | 30 + tools/playbooks/dsvm/post.yaml | 4 + tools/playbooks/dsvm/pre.yaml | 9 + tools/playbooks/dsvm/run.yaml | 11 + tools/playbooks/multinode_setup/common_config.yaml | 75 ++ .../multinode_setup/configure_loopback.yaml | 64 ++ tools/playbooks/multinode_setup/make_rings.yaml | 126 ++++ tools/playbooks/multinode_setup/pre.yaml | 8 + tools/playbooks/multinode_setup/run.yaml | 42 ++ .../templates/make_multinode_rings.j2 | 38 + .../multinode_setup/templates/rc.local.j2 | 8 + .../multinode_setup/templates/test.conf.j2 | 122 ++++ .../install_dependencies.yaml | 24 - .../saio_single_node_setup/setup_saio.yaml | 2 +- tox.ini | 33 +- 200 files changed, 8007 insertions(+), 3563 deletions(-)