We are satisfied to announce the release of:
swift 2.19.0: OpenStack Object Storage
This release is part of the rocky release series.
Download the package from:
https://tarballs.openstack.org/swift/
For more details, please see below.
2.19.0 ^^^^^^
New Features
* TempURLs now support IP range restrictions. Please see https://docs.openstack.org/swift/latest/middleware.html#client-usage for more information on how to use this additional restriction.
* Add support for multiple root encryption secrets for the trivial and KMIP keymasters. This allows operators to rotate encryption keys over time without needing to re-encrypt all existing data in the cluster. Please see the included sample config files for instructions on how to multiple encryption keys.
* The object updater now supports two configuration settings: "concurrency" and "updater_workers". The latter controls how many worker processes are spawned, while the former controls how many concurrent container updates are performed by each worker process. This should speed the processing of async_pendings.
On upgrade, a node configured with concurrency=N will still handle async updates N-at-a-time, but will do so using only one process instead of N.
If you have a config file like this:
[object-updater] concurrency = <N>
and you want to take advantage of faster updates, then do this:
[object-updater] concurrency = 8 # the default; you can omit this line updater_workers = <N>
If you want updates to be processed exactly as before, do this:
[object-updater] concurrency = 1 updater_workers = <N>
* When listing objects in a container in json format, static large objects (SLOs) will now include an additional new "slo_etag" key that matches the etag returned when requesting the SLO. The existing "hash" key remains unchanged as the MD5 of the SLO manifest. Text and XML listings are unaffected by this change.
* Log deprecation warnings for "run_pause". This setting was deprecated in Swift 2.4.0 and is replaced by "interval". It may be removed in a future release.
* Object reconstructor logs are now prefixed with information about the specific worker process logging the message. This makes reading the logs and understanding the messages much simpler.
* Lower bounds of dependencies have been updated to reflect what is actually tested.
* SSYNC replication mode now removes as much of the directory structure as possible as soon at it observes that the directory is empty. This reduces the work needed for subsequent replication passes.
* The container-updater now reports zero objects and bytes used for child DBs in sharded containers. This prevents double-counting in utilization reports.
* Add fallocate_reserve to account and container servers. This allows disks shared between account/container and object rings to avoid getting 100% full. The default value of 1% matches the existing default on object servers.
* Added an experimental "swift-ring-composer" CLI tool to build composite rings.
* Added an optional "read_only" middleware to make an entire cluster or individual accounts read only.
* Fixed a bug where zero-byte PUTs would not work properly with "If- None-Match: *" conditional requests.
* ACLs now work with unicode in user/account names.
* COPY now works with unicode account names.
* Improved S3 API compatibility.
* Lock timeouts in the container updater are now logged at INFO level, not ERROR.
* Various other minor bug fixes and improvements.
Changes in swift 2.18.0..2.19.0 -------------------------------
9bd84be authors/changelog for 2.19.0 release c63d5c0 Fix tempurl's example code to be in the code-block 7ca1a67 Add debugging info to SignatureDoesNotMatch responses 0dc1b62 Multi-key KMIP keymaster 2722e49 Add support for multiple root encryption secrets fc04dc1 Attempt to fix pep8 a7c5ca0 Fix locking in swift-recon-cron 51b885b s3_acl: Require swift_owner authz to create buckets bd640cd Fix the deletion of non-existent keys 5dca610 Fix up test_static_web_pretend_to_be_giant_json 0887f09 Update saio sample config files 03c1d61 Imported Translations from Zanata 7895718 Fix keymmaster_conf_section typo dcbdcb8 Disallow uploads with x-amz-tagging headers set b957940 Add probetests to gate jobs 79539ea Disallow aws-chunked uploads 86d78f6 imported some docs from the old user-guide 6a2bd0f improve prefix/delimiter api docs 6f920fd Add some reno artifacts to .gitignore 7fd5a12 Stop holding on to sys.exc_info tuples quite so much 3c92e3c Move keymaster_config_path parsing out of _get_root_secret 2f9bd85 Define keymaster log routes on the class ba8447e add lower-constraints job 84758d9 make probe tests voting in the gate c3d83ba Remove some unnecessary SkipTests 6a03990 Remove contentdir hack 60f1d34 Lower the required version of ipaddress 8e651a2 Add fallocate_reserve to account and container servers. 0cd42a2 Check other params preserved when slo_etag is extracted f8b9c24 Add unittest for slo_etag fa4c30d Include s3api schemas in sdists c4c98eb Include SLO ETag in container updates 4409da6 Update doc building instructions in Development Guidelines 2f4f2dd Add Alpine Linux suuprt to bindep 9a7b46e swift-ring-builder shows hint about composite builder file 7bacd53 Be consistent about capitalizing Swift in the README f27f1ae Make docs link consistent 26b20ee IP Range restrictions in temp urls 1951dc7 Add keymaster to fetch root secret from KMIP service 91a8cd2 PUT+POST: break out putter specific test classes 8bfb97a Content-Length enforcement fixups 783c7f6 py3: port proxy/controllers/info.py d03fc9b swob: Stop auto-encoding unicode bodies 600521c Describe separate keymaster config file in docs 108304e Improve README doc building instructions 8918615 Update README related to documentation building 7ae2cc4 Address some review comments 158d4c2 swob.Match: remove quotes when checking __contains__ efcf7e6 swob: Fix up some WSGI string business 01fb3ef Log unexpected errors when trying to clean up empty dirs 306f9a1 py3: port domain_remap tests 081ff36 py3: port crossdomain 50ea4e1 py3: port healthcheck 0edfb87 reorganized existing ansible playbooks 1fa4b9e Remove unnecessary loop in test_keymaster.py f0e4685 fix misspelling of 'server' 5652dec container-updater: Always report zero objects/bytes used for shards a5a6a27 rename swob.str_from_wsgi -> swob.wsgi_to_str 3920b88 py3: port catch_errors e0b4eb2 py3: fix test_proxy_logging.py 5b79ce9 Reverse shard ranges later 1318bac py36: Fix test_get_logger_sysloghandler_plumbing ba0b3c3 Add clarifying comment about surrogate check f192f51 Have check_drive raise ValueError on errors 2b19c26 py3: Port common/swob 0694e51 py3: Port common/constraints 2637fec Add py36 testenv f21bbe0 specify test selector with py3 packages in bindep a59c5e3 Resolve TODO's in test/probe/test_sharder.py 554c07a Fix the example webhook middleware a1cb919 Use valid eventlet logger method 642d874 Fix test_wsgi.py tests failing with eventlet <=0.18.4 4a0afa9 Enforce Content-Length in catch_errors b08355e Rename test_except.py -> test_catch_errors.py ac69a71 s3api: Stop debug-logging the entire wsgi environment 049dc0c Blacklist eventlet 0.23.0 572b6be Set lower bounds on all requirements and test-requirements 8e89271 Tighten up staticweb redirect test b39a88c func tests: Rename storage_url to storage_path 7042d2c Use path_qs instead of reinventing it f6a436a Only try to fetch or sync shard ranges if the remote supports sharding a718cb3 py3: fix up some NameErrors 6b626f2 Experimental swift-ring-composer CLI to build composite rings 026f706 rectify 'a integer number' to 'an integer number' 2a2ce04 rectify 'a invalid version' to 'an invalid version' d5c532a object-updater: add concurrent updates a04234d Functional tests for domain_remap middleware 65e1de2 Fix shard_max_row in ContainerBroker.get_replication_info() 2ee8363 rectify 'a expirting object' to 'an expiring object' f2295a6 Fix typo in sharding docs dc8d1c9 Get rid of tpool_reraise b3ce3c9 func tests: Drop clone_conn arg to Connection fcd7a05 Fix up insecure behavior for functional tests 0318e7c Rearrange some error handling to reduce indent beb014b Read-only middleware fixups e726473 Fix typos replace a OpenStack bucket with an OpenStack bucket 284da56 Fix grammar errors replace a object with an object 8b44c9d Allow domain remap storage_domain to be configured for func tests caa3c67 functests: Clean up objects better c88d4d7 Make COPY work with unicode account names 9d0162a Make ACLs work with Unicode in user/account names 7cec5a8 tempauth: Send back url-encoded account names 9ef2a82 copy: Stop hanging per-request data on middleware instance 1424ff7 Refactor object updater's async-pending finder ecf4755 Make final stats dump after reconstructor runs once 76344e5 Remove redundant elif in tempauth 2044673 Improve user logging 5acefa3 made changelog section breaks more consistent e748ef4 Verify diff stat is unchanged when syncing only shard ranges 5078bb1 Added S3 compatibility information to docs bb9e427 Add detail to container server GET docstring 7d76017 Fix marker and end_marker descriptions in API ref 076f20b removed specific reference to two core reviewers 47aff72 Log updater stats once per pass, not per policy fb3b43f Added PyPI validation check for the readme. 5d601b7 Adds read_only middleware a954f59 container-updater: log LockTimeout exceptions at INFO, not ERROR 553bef0 Remove unhelpful log message in copy middleware 31623f7 Catch and log errors from process_container, then keep updating 7b5c30d Update SERVER_ADDR/SERVER_PORT from PROXY protocol 854db51 Fix common/test_utils.py on Python 3.5.4+ 844bba8 Remove some Python 2.6 compatibility code in ring 7a76778 Use X-Timestamp when checking object expiration 0702d4d Stop overriding _in_sync in container replicator a19548b Remove empty part dirs during ssync replication 26538d3 Make multiprocess reconstructor's logs more readable. f8419bf Improve path handling in proxy_logging 748b29e Make If-None-Match:* work properly with 0-byte PUTs e0d1869 Fix suffix-byte-range responses for zero-byte EC objects. c44d047 Create policy_stat table in auditor if missing b0ab7eb Log deprecation warnings for run_pause f789af6 display swift services in apache2 b29eb13 Add assertion to test_consolidate_hashes_raises_exception
Diffstat (except docs and test files) -------------------------------------
.gitignore | 2 + .zuul.yaml | 14 +- AUTHORS | 8 + CHANGELOG | 130 +- CONTRIBUTING.rst | 2 +- MANIFEST.in | 1 + README.rst | 17 +- api-ref/source/parameters.yaml | 25 +- bandit.yaml | 2 +- bin/swift-recon-cron | 26 +- bin/swift-ring-composer | 22 + bindep.txt | 21 +- .../pseudo-hierarchical-folders-directories.rst | 155 ++ etc/account-server.conf-sample | 7 + etc/container-server.conf-sample | 7 + etc/keymaster.conf-sample | 20 + etc/object-server.conf-sample | 9 +- etc/proxy-server.conf-sample | 40 + examples/apache2/account-server.template | 2 +- examples/apache2/container-server.template | 2 +- examples/apache2/object-server.template | 2 +- examples/apache2/proxy-server.template | 2 +- lower-constraints.txt | 89 + .../install_dependencies.yaml | 50 - playbooks/saio_single_node_setup/make_rings.yaml | 29 - playbooks/saio_single_node_setup/setup_saio.yaml | 174 -- .../saio_single_node_setup/templates/rc.local.j2 | 8 - .../notes/2_19_0_release-3e6ee3e6a1fcc6bb.yaml | 103 ++ .../locale/en_GB/LC_MESSAGES/releasenotes.po | 325 +++- .../locale/ko_KR/LC_MESSAGES/releasenotes.po | 75 - requirements.txt | 5 +- setup.cfg | 6 + swift/account/backend.py | 7 +- swift/account/reaper.py | 7 +- swift/account/server.py | 40 +- swift/cli/ringbuilder.py | 9 +- swift/cli/ringcomposer.py | 183 ++ swift/common/constraints.py | 110 +- swift/common/db_replicator.py | 42 +- swift/common/exceptions.py | 4 + swift/common/internal_client.py | 2 +- swift/common/middleware/acl.py | 11 +- swift/common/middleware/catch_errors.py | 70 +- swift/common/middleware/copy.py | 40 +- swift/common/middleware/crossdomain.py | 2 +- swift/common/middleware/crypto/crypto_utils.py | 26 +- swift/common/middleware/crypto/decrypter.py | 150 +- swift/common/middleware/crypto/encrypter.py | 11 +- swift/common/middleware/crypto/keymaster.py | 197 +- swift/common/middleware/crypto/kmip_keymaster.py | 158 ++ swift/common/middleware/crypto/kms_keymaster.py | 27 +- swift/common/middleware/dlo.py | 2 + swift/common/middleware/domain_remap.py | 2 +- swift/common/middleware/gatekeeper.py | 11 +- swift/common/middleware/healthcheck.py | 4 +- swift/common/middleware/list_endpoints.py | 2 +- swift/common/middleware/proxy_logging.py | 15 +- swift/common/middleware/read_only.py | 123 ++ swift/common/middleware/recon.py | 22 +- swift/common/middleware/s3api/acl_handlers.py | 10 +- .../common/middleware/s3api/controllers/bucket.py | 4 +- .../middleware/s3api/controllers/multi_delete.py | 5 +- .../middleware/s3api/controllers/multi_upload.py | 6 +- swift/common/middleware/s3api/controllers/obj.py | 10 +- swift/common/middleware/s3api/etree.py | 9 +- swift/common/middleware/s3api/s3api.py | 1 - swift/common/middleware/s3api/s3request.py | 37 +- swift/common/middleware/s3api/utils.py | 8 +- swift/common/middleware/slo.py | 92 +- swift/common/middleware/tempauth.py | 23 +- swift/common/middleware/tempurl.py | 101 +- swift/common/ring/ring.py | 5 +- swift/common/swob.py | 133 +- swift/common/utils.py | 78 +- swift/common/wsgi.py | 69 +- swift/container/backend.py | 27 +- swift/container/reconciler.py | 8 + swift/container/replicator.py | 25 +- swift/container/server.py | 108 +- swift/container/updater.py | 28 +- swift/locale/de/LC_MESSAGES/swift.po | 18 +- swift/locale/en_GB/LC_MESSAGES/swift.po | 42 +- swift/locale/es/LC_MESSAGES/swift.po | 20 +- swift/locale/fr/LC_MESSAGES/swift.po | 18 +- swift/locale/it/LC_MESSAGES/swift.po | 18 +- swift/locale/ja/LC_MESSAGES/swift.po | 22 +- swift/locale/ko_KR/LC_MESSAGES/swift.po | 16 +- swift/locale/pt_BR/LC_MESSAGES/swift.po | 16 +- swift/locale/ru/LC_MESSAGES/swift.po | 18 +- swift/locale/tr_TR/LC_MESSAGES/swift.po | 16 +- swift/locale/zh_CN/LC_MESSAGES/swift.po | 16 +- swift/locale/zh_TW/LC_MESSAGES/swift.po | 16 +- swift/obj/diskfile.py | 133 +- swift/obj/mem_diskfile.py | 19 +- swift/obj/reconstructor.py | 55 +- swift/obj/replicator.py | 40 +- swift/obj/server.py | 20 +- swift/obj/updater.py | 168 +- swift/proxy/controllers/base.py | 22 +- swift/proxy/controllers/info.py | 2 +- swift/proxy/controllers/obj.py | 69 +- swift/proxy/server.py | 4 +- test-requirements.txt | 12 +- test/functional/__init__.py | 14 +- test/functional/s3api/test_bucket.py | 6 +- test/functional/s3api/test_object.py | 11 +- test/functional/swift_test_client.py | 96 +- test/functional/test_container.py | 6 +- test/functional/test_domain_remap.py | 177 ++ test/functional/test_object.py | 6 +- test/functional/test_slo.py | 85 +- test/functional/test_staticweb.py | 54 +- test/functional/test_symlink.py | 50 +- test/functional/test_tempurl.py | 4 - test/probe/test_object_conditional_requests.py | 79 + test/probe/test_sharder.py | 134 +- test/sample.conf | 5 + test/unit/__init__.py | 27 +- test/unit/account/test_backend.py | 76 +- test/unit/account/test_server.py | 86 + test/unit/cli/test_ringbuilder.py | 50 +- test/unit/cli/test_ringcomposer.py | 195 ++ .../common/middleware/crypto/crypto_helpers.py | 31 +- .../common/middleware/crypto/test_crypto_utils.py | 47 +- .../common/middleware/crypto/test_decrypter.py | 196 +- .../common/middleware/crypto/test_encrypter.py | 11 +- .../common/middleware/crypto/test_encryption.py | 39 +- .../common/middleware/crypto/test_keymaster.py | 251 ++- .../middleware/crypto/test_kmip_keymaster.py | 270 +++ .../common/middleware/crypto/test_kms_keymaster.py | 68 +- test/unit/common/middleware/s3api/__init__.py | 11 +- test/unit/common/middleware/s3api/helpers.py | 16 +- test/unit/common/middleware/s3api/test_bucket.py | 6 + .../common/middleware/s3api/test_multi_delete.py | 14 +- test/unit/common/middleware/s3api/test_obj.py | 38 +- test/unit/common/middleware/s3api/test_s3api.py | 27 +- test/unit/common/middleware/s3api/test_service.py | 5 +- test/unit/common/middleware/test_catch_errors.py | 199 ++ test/unit/common/middleware/test_crossdomain.py | 32 +- test/unit/common/middleware/test_dlo.py | 15 + test/unit/common/middleware/test_domain_remap.py | 80 +- test/unit/common/middleware/test_except.py | 142 -- test/unit/common/middleware/test_gatekeeper.py | 4 +- test/unit/common/middleware/test_healthcheck.py | 12 +- .../unit/common/middleware/test_listing_formats.py | 15 +- test/unit/common/middleware/test_proxy_logging.py | 113 +- test/unit/common/middleware/test_read_only.py | 237 +++ test/unit/common/middleware/test_slo.py | 55 +- .../common/middleware/test_subrequest_logging.py | 11 +- test/unit/common/middleware/test_tempauth.py | 30 +- test/unit/common/middleware/test_tempurl.py | 214 ++- test/unit/common/test_constraints.py | 180 +- test/unit/common/test_db_replicator.py | 7 +- test/unit/common/test_swob.py | 356 ++-- test/unit/common/test_utils.py | 85 +- test/unit/common/test_wsgi.py | 98 +- test/unit/container/test_backend.py | 35 + test/unit/container/test_replicator.py | 139 +- test/unit/container/test_server.py | 89 +- test/unit/container/test_updater.py | 121 +- test/unit/obj/test_diskfile.py | 180 +- test/unit/obj/test_reconstructor.py | 221 ++- test/unit/obj/test_replicator.py | 43 +- test/unit/obj/test_server.py | 265 +-- test/unit/obj/test_ssync.py | 30 +- test/unit/obj/test_updater.py | 89 +- test/unit/proxy/controllers/test_info.py | 10 +- test/unit/proxy/controllers/test_obj.py | 1900 ++++++++++---------- test/unit/proxy/test_server.py | 20 +- .../install_dependencies.yaml | 24 + .../saio_single_node_setup/make_rings.yaml | 29 + .../saio_single_node_setup/setup_saio.yaml | 174 ++ .../saio_single_node_setup/templates/rc.local.j2 | 8 + tox.ini | 24 +- 212 files changed, 9420 insertions(+), 3414 deletions(-)
Requirements updates --------------------
diff --git a/requirements.txt b/requirements.txt index c115113..df5d26b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +6 @@ dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE -eventlet>=0.17.4 # MIT +eventlet>=0.17.4,!=0.23.0 # MIT @@ -10 +10 @@ PasteDeploy>=1.3.3 -lxml +lxml>=3.4.1 @@ -15,0 +16 @@ cryptography!=2.0,>=1.6 # BSD/Apache-2.0 +ipaddress>=1.0.16;python_version<'3.3' # PSF diff --git a/test-requirements.txt b/test-requirements.txt index 5f515bd..cf97918 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8,2 +8,2 @@ coverage>=3.6 # Apache-2.0 -nose # LGPL -nosexcover # BSD +nose>=1.3.7 # LGPL +nosexcover>=1.0.10 # BSD @@ -13 +13 @@ mock>=2.0 # BSD -python-swiftclient +python-swiftclient>=3.2.0 @@ -16,2 +16,2 @@ reno>=1.8.0 # Apache-2.0 -python-openstackclient -boto +python-openstackclient>=3.12.0 +boto>=2.32.1 @@ -23,0 +24,2 @@ bandit>=1.1.0 # Apache-2.0 + +docutils>=0.11 # OSI-Approved Open Source, Public Domain