We are chuffed to announce the release of: swift 2.22.0: OpenStack Object Storage This release is part of the train release series. The source is available from: https://opendev.org/openstack/swift Download the package from: https://tarballs.openstack.org/swift/ Please report issues through: https://bugs.launchpad.net/swift/+bugs For more details, please see below. 2.22.0 ^^^^^^ New Features ************ * Experimental support for Python 3.6 and 3.7 is now available. Note that this requires "eventlet>=0.25.0". All unit tests pass, and running functional tests under Python 2 will pass against services running under Python 3. Expect full support in the next minor release. * Log formats are now more configurable and include support for anonymization. See the "log_msg_template" option in "proxy- server.conf" and the Swift documentation for more information. * Added an operator tool, "swift-container-deleter", to asynchronously delete some or all objects in a container using the object expirers. * Swift-all-in-one Docker images are now built and published to https://hub.docker.com/r/openstackswift/saio. These are intended for use as development targets, but will hopefully be useful as a starting point for other work involving containerizing Swift. Upgrade Notes ************* * The "object-expirer" may now be configured in "object- server.conf". This is in anticipation of a future change to allow the "object- expirer" to be deployed on all nodes that run the "object-server". * **Dependency updates**: we've increased our minimum supported version of "cryptography" to 2.0.2 and "netifaces" to 0.8. This is largely due to the difficulty of continuing to test with the old versions. If running Swift under Python 3, "eventlet" must be at least 0.25.0. Bug Fixes ********* * Correctness improvements * The "proxy-server" now ignores 404 responses from handoffs without databases when deciding on the correct response for account and container requests. * Object writes to a container whose existence cannot be verified now 503 instead of 404. * Sharding improvements * The "container-replicator" now only attempts to fetch shard ranges if the remote indicates that it has shard ranges. Further, it does so with a timeout to prevent the process from hanging in certain cases. * The "proxy-server" now caches 'updating' shards, improving write performance for sharded containers. A new config option, "recheck_updating_shard_ranges", controls the cache time; set it to 0 to disable caching. * The "container-replicator" now correctly enqueues "container- reconciler" work for sharded containers. * S3 API improvements * Unsigned payloads work with v4 signatures once more. * Multipart upload parts may now be copied from other multipart uploads. * CompleteMultipartUpload requests with a "Content-MD5" now work. * "Content-Type" can now be updated when copying an object. * Fixed v1 listings that end with a non-ASCII object name. * Background corruption-detection improvements * Detect and remove invalid entries from "hashes.pkl" * When object path is not a directory, just quarantine it, rather than the whole suffix. * Various other minor bug fixes and improvements. Changes in swift 2.21.0..2.22.0 ------------------------------- a48ac28ef Authors/changelog for 2.22.0 a1af3811a sharding: Cache shard ranges for object writes 0ae1ad63c Update auth_url in install docs f4bb1bea2 reconciler: Enqueue right work for shard containers c1d170225 functests: Make test_PUT_metadata less flakey 9d1b74974 py3: port staticweb and domain_remap func tests 38a24571a functests: make container creation less flakey in test_object 4c4bd778e container-replicator: Add a timeout for get_shard_ranges 345f577ff s3token: fix conf option name 76fde8926 py3: Be able to read and write non-ASCII headers 4f7c44a9d Add information about secret_cache_duration to sample config 072b40451 s3api: add unit test case for s3acl c46b88ab7 Fix invalid assert states 88cec2c39 s3api: Put more information in the str() for InternalError e02b9b798 Use get_container_info to check existence before container PUT 044c91987 More tests for 404 handoff skipping 563e1671c Return 503 when primary containers can't respond 82169ead1 Don't handle object without container 55b04d16c Run all func tests under py2 against services running py3 39a54fecd py3: add swift-dsvm-functional-py3 job ff4459ed6 Move call to global_conf_callback after loadapp() 9f1ef3563 py3: Port the tempurl middleware a57085edc py3: port the test of locale b10f4bae2 func tests: tolerate more 404s when deleting 4ab3706b7 Remove nobarrier option 876ce231d py3: increase _MAXHEADERS for eventlet, too 557335465 Move calls to self.app outside of error handling 6374734d7 Stop manually installing eventlet for our py3 tests 014358018 py3: Decode some things in s3api ef664700d py3: Specify an encoding when loading db.pending pickles e6c971b7e Stop sending non-UTF-8 prefixes in func tests ccd3b890c Enable s3api in saio docker container ef8818a63 Fix up how we memcache on py3 0f46ace8a Fix :param: in docstring aa2f1db1b Ensure get_*_info keys are native strings b7b92b97b Bump up minimum cryptography version 668e45af6 Update mailmap for Kota Tsuyuzaki 06e4533c2 py3: Fix header_to_environ_key e8e7106d1 py3: port obj/reconstructor tests dca658103 Fix swift with python <2.7.9 d9cafca24 py3: port ssync a122b720e Dockerfile cleanup 17fd72570 Installing liberasurecode from Alpine Linux repos instead of building from source. 2eb245168 Increase some middleware test coverage 98637dc1e Bump openstackdocstheme to 1.30.0 79ab3a490 Note s3api compatability requirements in sample.conf eaf180c68 Fix a few Sphinx warnings 37fa12cd8 Do not sync suffixes when remote rejects reconstructor sync c9e78d15e Remove invalid dict entries from hashes.pkl 2e35376c6 py3: symlink follow-up 4f9595f11 py3: port symlink middleware 2c4f28405 Modify the version path URL of the configuration file ff04ef05c Rework private-request-method interface e28bc3c29 Fix incorrect setting of symlink_target_account 04c8015b8 Add more tests for async expirer e22960fd7 s3api: Fix ETag when copying a MU part from another MU 40bd9f16a py3: port diskfile 8868e2cd5 Clean up test_write_read_metadata 83d016199 Add operator tool to async-delete some or all objects in a container 4927b1f29 Specify pickle protocol in REPLICATE() 61a4567a8 Controller install Debian missing proxy-server.conf configuration 6008b0a83 Fix quarantine when object path is not a directory d4e794045 Start transition to boto3 in s3api tests. 53f8ccb4a container: Unify storage-policy error paths 350bc8cbe s3api: stop adding etag quotes when there are already etag quotes 82e446a8a s3api: Allow clients to upload with UNSIGNED-PAYLOAD bd8c3067b py3: slo ad034023f Be explicit that a code path is just for py2 add5a411a Make build/upload/promote image jobs non-voting bec2cccd4 py3: port expirer 8b6438137 Set Content-Type with s3api metadata updates. 5cd2a2ac2 py3: bulk -- alternate approach 89c9c6f0b Have a separate s3api functional test suite 51e415b74 py3: port proxy/test_mem_server.py 8b3d0a6c6 py3: finish porting proxy/test_server.py 4f4456561 Add py37 func test job 506279235 Get functional/tests.py running under py3 259224f00 py3: port unit/proxy/test_server.py dade632b0 S3Api: handle non-ASCII markers in v1 listings. 032cf3b3b Update docker secret. 37e257913 Add Dockerfile to build a SAIO container image 211cf3983 Require logger passed to broker in unit tests for account-reaper 6d63297f0 [Trivial fix]Remove unnecessary slash 767872210 py3: port unit/proxy/test_sysmeta.py fed0204e8 py3: Make StoragePolicy objects hashable b8284538b py3: start porting for unit/proxy/test_server.py 34bd4f7fa Clarify usage of dequeue_from_legacy option 9290f29e1 Wait longer for log lines in unit test 443f029a5 Enable to configure object-expirer in object-server.conf 1e4cdca33 Fix ostestr invocation f5b472cdc Fix wrong tox env name from func-post-ec to func-ec 359d136c2 Unify auth_host/port/ssl/prefix func test options into auth_uri a239f4ec3 Remove py35 unit test job a4cc35337 Make log format for requests configurable 3a9f3f841 py3: port s3api c0ae48ba9 py3: Stop using stdlib's putrequest(); it only does ASCII 93b49c5e4 py3: Be able to parse non-RFC-compliant request lines 4b2ac606b Pass logger instances to AccountBroker/ContainerBroker 58352b285 s3api multi upload test clean up c51db382c s3api: Block ETag header from CompleteMultipartUpload requests 32bf43990 Replace git.openstack.org URLs with opendev.org URLs 89eced960 OpenDev Migration Patch 9468e28cc Make test_read_write_acl_listings more robust b6ebabee7 Clean up dlo unit tests 893acffbc py3: port dlo 7cb276da4 Move from py35 to py37 for gating py3 jobs 65d3cb75f Use assertIn to check whether substring is in str e5ff405ec Make staticweb return URL-encoded Location headers 049e56a5d Remove our urlparse wrapper 5b40f69ed Add tests for unexpected names in part dir b86bc5160 py3: port staticweb middleware 621e92488 Update openstack repo for libec install 08d59ebcd Change partition name from string to integer e5e22ebeb Make symlink work with Unicode account names 52bc11f09 Use labels to mount filesystems in the guide 0294836d9 py3: test/unit/common/middleware/s3api/test_s3token.py adc826f37 Clean up more things before running unit tests 9d9f8b0ff add more detail to s3api test failure e3e79a6e4 Remove "gross hack" for py3pep8 env; run releasenotes under py3 6c93c5768 Run docs tox env under py3 ffe51501c api-ref: add link to SLO docs from multipart-manifest=put param 9b694a332 s3token: Add an underscore to the end of reseller_prefix if not present 8f23dbad6 Make reconciler test more stable in the gate 89e5927f7 Fix mocking time 50715acb1 Update master for stable/stein 10cb205fe Fix lower-constraint deps handling 8e045d0de versioned_writes: stop trying to catch HTTPPreconditionFailed 771963c92 Increase node_timeout in gate a4298fb3a s3api: Fix get_container_info missing some sysmeta info. 29b877de7 s3api: fix partition between S3 and Swift headers Diffstat (except docs and test files) ------------------------------------- .dockerignore | 11 + .gitreview | 2 +- .mailmap | 4 + .zuul.yaml | 152 +- AUTHORS | 15 +- CHANGELOG | 77 +- CONTRIBUTING.rst | 2 +- Dockerfile | 71 + api-ref/source/conf.py | 5 + api-ref/source/parameters.yaml | 2 + .../finalize-installation-ubuntu-debian.rst | 2 +- .../install/storage-install-ubuntu-debian.rst | 10 +- .../15_install_python_and_pip_src.sh | 57 + .../rootfs/etc/socklog.rules/swift_account_server | 3 + .../etc/socklog.rules/swift_container_server | 3 + .../rootfs/etc/socklog.rules/swift_object_server | 3 + etc/account-server.conf-sample | 11 + etc/container-server.conf-sample | 11 + etc/object-server.conf-sample | 77 + etc/proxy-server.conf-sample | 38 +- lower-constraints.txt | 9 +- .../notes/2_22_0_release-f60d29508b3c1283.yaml | 87 + releasenotes/source/index.rst | 2 + releasenotes/source/stein.rst | 6 + requirements.txt | 14 +- setup.cfg | 1 + swift/account/auditor.py | 2 +- swift/account/reaper.py | 3 +- swift/account/server.py | 8 +- swift/cli/container_deleter.py | 174 ++ swift/cli/recon.py | 3 +- swift/common/base_storage_server.py | 6 +- swift/common/bufferedhttp.py | 31 +- swift/common/db.py | 6 +- swift/common/db_replicator.py | 11 +- swift/common/manager.py | 69 +- swift/common/memcached.py | 12 +- swift/common/middleware/acl.py | 4 +- swift/common/middleware/bulk.py | 117 +- swift/common/middleware/copy.py | 3 + swift/common/middleware/crypto/decrypter.py | 3 + swift/common/middleware/dlo.py | 54 +- swift/common/middleware/formpost.py | 10 +- swift/common/middleware/list_endpoints.py | 11 +- swift/common/middleware/listing_formats.py | 4 + swift/common/middleware/proxy_logging.py | 149 +- swift/common/middleware/ratelimit.py | 3 +- .../common/middleware/s3api/controllers/bucket.py | 28 +- .../middleware/s3api/controllers/multi_upload.py | 77 +- swift/common/middleware/s3api/controllers/obj.py | 2 +- .../common/middleware/s3api/controllers/service.py | 4 +- swift/common/middleware/s3api/etree.py | 4 +- swift/common/middleware/s3api/s3api.py | 10 +- swift/common/middleware/s3api/s3request.py | 122 +- swift/common/middleware/s3api/s3response.py | 24 +- swift/common/middleware/s3api/s3token.py | 9 +- .../middleware/s3api/schema/list_bucket_result.rng | 4 +- swift/common/middleware/s3api/subresource.py | 14 +- swift/common/middleware/s3api/utils.py | 6 +- swift/common/middleware/slo.py | 144 +- swift/common/middleware/staticweb.py | 43 +- swift/common/middleware/symlink.py | 32 +- swift/common/middleware/tempurl.py | 20 +- swift/common/middleware/versioned_writes.py | 75 +- swift/common/middleware/x_profile/html_viewer.py | 3 +- swift/common/storage_policy.py | 3 + swift/common/swob.py | 67 +- swift/common/utils.py | 216 +- swift/common/wsgi.py | 66 +- swift/container/auditor.py | 2 +- swift/container/reconciler.py | 4 +- swift/container/replicator.py | 7 +- swift/container/server.py | 51 +- swift/container/sharder.py | 5 +- swift/container/sync.py | 10 +- swift/obj/diskfile.py | 58 +- swift/obj/expirer.py | 115 +- swift/obj/mem_diskfile.py | 16 +- swift/obj/reconstructor.py | 15 +- swift/obj/replicator.py | 4 +- swift/obj/server.py | 14 +- swift/obj/ssync_receiver.py | 52 +- swift/obj/ssync_sender.py | 80 +- swift/proxy/controllers/base.py | 185 +- swift/proxy/controllers/container.py | 35 +- swift/proxy/controllers/obj.py | 11 +- swift/proxy/server.py | 35 +- test-requirements.txt | 2 + test/functional/__init__.py | 51 +- test/functional/s3api/__init__.py | 23 +- test/functional/s3api/s3_test_client.py | 62 +- test/functional/s3api/test_bucket.py | 557 ++--- test/functional/s3api/test_multi_upload.py | 95 +- test/functional/swift_test_client.py | 153 +- test/functional/test_container.py | 16 +- test/functional/test_dlo.py | 25 +- test/functional/test_domain_remap.py | 25 +- test/functional/test_object.py | 2 +- test/functional/test_slo.py | 55 +- test/functional/test_staticweb.py | 96 +- test/functional/test_symlink.py | 260 +- test/functional/test_tempurl.py | 152 +- test/functional/test_versioned_writes.py | 9 +- test/probe/common.py | 8 +- test/probe/test_container_failures.py | 5 +- test/probe/test_sharder.py | 23 +- test/s3api/__init__.py | 142 ++ test/s3api/test_service.py | 91 + test/sample.conf | 27 +- test/unit/__init__.py | 32 +- test/unit/account/test_auditor.py | 3 +- test/unit/account/test_reaper.py | 56 +- test/unit/account/test_server.py | 14 +- test/unit/cli/test_container_deleter.py | 278 +++ .../common/middleware/crypto/test_crypto_utils.py | 8 + .../middleware/crypto/test_kmip_keymaster.py | 24 +- test/unit/common/middleware/s3api/__init__.py | 2 + test/unit/common/middleware/s3api/helpers.py | 7 + test/unit/common/middleware/s3api/test_acl.py | 14 +- test/unit/common/middleware/s3api/test_bucket.py | 114 +- test/unit/common/middleware/s3api/test_etree.py | 15 +- .../common/middleware/s3api/test_multi_delete.py | 29 +- .../common/middleware/s3api/test_multi_upload.py | 270 +- test/unit/common/middleware/s3api/test_obj.py | 64 +- test/unit/common/middleware/s3api/test_s3_acl.py | 9 +- test/unit/common/middleware/s3api/test_s3api.py | 58 +- .../unit/common/middleware/s3api/test_s3request.py | 114 +- .../common/middleware/s3api/test_s3response.py | 2 + test/unit/common/middleware/s3api/test_s3token.py | 17 +- test/unit/common/middleware/s3api/test_service.py | 2 +- test/unit/common/middleware/test_bulk.py | 144 +- test/unit/common/middleware/test_crossdomain.py | 2 +- test/unit/common/middleware/test_dlo.py | 117 +- test/unit/common/middleware/test_proxy_logging.py | 42 + test/unit/common/middleware/test_recon.py | 2 +- test/unit/common/middleware/test_slo.py | 402 +-- test/unit/common/middleware/test_staticweb.py | 96 +- test/unit/common/middleware/test_symlink.py | 131 +- test/unit/common/middleware/test_tempurl.py | 312 +-- test/unit/common/test_db.py | 17 + test/unit/common/test_internal_client.py | 2 +- test/unit/common/test_manager.py | 117 + test/unit/common/test_memcached.py | 4 + test/unit/common/test_storage_policy.py | 16 +- test/unit/common/test_swob.py | 38 +- test/unit/common/test_utils.py | 143 +- test/unit/common/test_wsgi.py | 53 +- test/unit/container/test_auditor.py | 2 +- test/unit/container/test_reconciler.py | 6 +- test/unit/container/test_replicator.py | 195 +- test/unit/container/test_server.py | 128 +- test/unit/container/test_sync.py | 41 +- test/unit/obj/test_auditor.py | 4 +- test/unit/obj/test_diskfile.py | 382 +-- test/unit/obj/test_expirer.py | 201 +- test/unit/obj/test_reconstructor.py | 154 +- test/unit/obj/test_replicator.py | 24 +- test/unit/obj/test_server.py | 91 +- test/unit/obj/test_ssync.py | 80 +- test/unit/obj/test_ssync_receiver.py | 432 ++-- test/unit/obj/test_ssync_sender.py | 268 +- test/unit/proxy/controllers/test_base.py | 51 +- test/unit/proxy/controllers/test_container.py | 16 +- test/unit/proxy/controllers/test_obj.py | 28 +- test/unit/proxy/test_server.py | 2569 ++++++++++++-------- test/unit/proxy/test_sysmeta.py | 31 +- test/unit/test_locale/eo.po | 5 + test/unit/test_locale/messages.mo | Bin 83 -> 181 bytes test/unit/test_locale/test_locale.py | 65 +- .../multinode_setup/configure_loopback.yaml | 2 +- tools/playbooks/multinode_setup/make_rings.yaml | 1 + .../saio_single_node_setup/setup_saio.yaml | 8 +- tools/test-setup.sh | 2 +- tox.ini | 114 +- 237 files changed, 9413 insertions(+), 4332 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index df5d26b69..609e540e6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,2 +5,2 @@ -dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE -eventlet>=0.17.4,!=0.23.0 # MIT +dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE +eventlet>=0.17.4,!=0.23.0 # MIT @@ -8 +8 @@ greenlet>=0.3.1 -netifaces>=0.5,!=0.10.0,!=0.10.1 +netifaces>=0.8,!=0.10.0,!=0.10.1 @@ -11 +11 @@ lxml>=3.4.1 -requests>=2.14.2 # Apache-2.0 +requests>=2.14.2 # Apache-2.0 @@ -13 +13 @@ six>=1.9.0 -xattr>=0.4 +xattr>=0.4;sys_platform!='win32' # MIT @@ -15,2 +15,2 @@ PyECLib>=1.3.1 # BSD -cryptography!=2.0,>=1.6 # BSD/Apache-2.0 -ipaddress>=1.0.16;python_version<'3.3' # PSF +cryptography>=2.0.2 # BSD/Apache-2.0 +ipaddress>=1.0.16;python_version<'3.3' # PSF diff --git a/test-requirements.txt b/test-requirements.txt index cf97918f0..ff86a01dd 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -17,0 +18,2 @@ boto>=2.32.1 +boto3>=1.9 +botocore>=1.12