We are pumped to announce the release of: swift 2.18.0: OpenStack Object Storage This release is part of the rocky release series. Download the package from: https://tarballs.openstack.org/swift/ For more details, please see below. 2.18.0 ^^^^^^ New Features * Added container sharding, an operator controlled feature that may be used to shard very large container databases into a number of smaller shard containers. This mitigates the issues with one large DB by distributing the data across multiple smaller databases throughout the cluster. Please read the full overview at https://do cs.openstack.org/swift/latest/overview_container_sharding.html * Provide an S3 API compatibility layer. The external "swift3" project has been imported into Swift's codebase as the "s3api" middleware. * Added "emergency mode" hooks in the account and container replicators. These options may be used to prioritize moving handoff partitions to primary locations more quickly. This helps when adding capacity to a ring. * Added "-d <devs>" and "-p <partitions>" command line options. * Added a handoffs-only mode. * Add a multiprocess mode to the object replicator. Setting the "replicator_workers" setting to a positive value N will result in the replicator using up to N worker processes to perform replication tasks. At most one worker per disk will be spawned. Worker process logs will have a bit of information prepended so operators can tell which messages came from which worker. The prefix is "[worker M/N pid=P] ", where M is the worker's index, N is the total number of workers, and P is the process ID. Every message from the replicator's logger will have the prefix * The object reconstructor will now fork all available worker processes when operating on a subset of local devices. * Add support for PROXY protocol v1 to the proxy server. This allows the Swift proxy server to log accurate client IP addresses when there is a proxy or SSL-terminator between the client and the Swift proxy server. Example servers supporting this PROXY protocol include stunnel, haproxy, hitch, and varnish. See the sample proxy server config file for the appropriate config setting to enable or disable this functionality. * In the ratelimit middleware, account whitelist and blacklist settings have been deprecated and may be removed in a future release. When found, a deprecation message will be logged. Instead of these config file values, set X-Account-Sysmeta- Global-Write- Ratelimit:WHITELIST and X-Account-Sysmeta-Global- Write- Ratelimit:BLACKLIST on the particular accounts that need to be whitelisted or blacklisted. System metadata cannot be added or modified by standard clients. Use the internal client to set sysmeta. * Add a "--drop-prefixes" flag to swift-account-info, swift- container- info, and swift-object-info. This makes the output between the three more consistent. * statsd error messages correspond to 5xx responses only. This makes monitoring more useful because actual errors (5xx) will not be hidden by common user requests (4xx). Previously, some 4xx responses would be included in timing information in the statsd error messages. * Truncate error logs to prevent log handler from running out of buffer. * Updated requirements.txt to match global exclusions and formatting. * tempauth user names now support unicode characters. * Various other minor bug fixes and improvements. Changes in swift 2.17.0..2.18.0 ------------------------------- fbb3b1f authors/changelog update for 2.18.0 release 0a964f8 add s3_acl option to s3api /info 36dbd38 Add s3api headers to allowed_headers by default 773b633 Change default sharding threshold to 1,000,000 objects c341c2c fix typos in comment 4189a11 catch lock fail exception in container updater 71b39e4 Remove un-used stub fbb8d7e Clarify that archive location headers should be URL-encoded d52a527 Clean out Python 2.6 leftovers from splice.py a6f7600 Remove our reimplemented logging.NullHandler 693d9a6 Always pass capitalize_response_headers=False to eventlet.wsgi.server() 661838d Add support for PROXY protocol v1 (only) 0aad950 Fix SLO delete for accounts with non-ASCII names. e4045fb Add unit tests for replicator sync_shard_ranges 3d3393c Fix typo 6714af8 Use maybe_get more 2e321d9 Add a helper context for optional connection args 4af57db Let make_db_file_path accept epoch=None f68dd3b Move handling of missing table outside of do_query 8486397 Clean up logging interpolation 8c386ff Make the decision between primary/handoff sets more obvious 0e3e7b9 Fix versioned writes error with url-encoded object name f237c58 Bump up timeout for swift-probetests-centos-7 37ee89e Avoid premature shrinking in sharder probe test 82314bd Test rsync error log truncation 0ee2a1d Simplify ContainerBroker._get_shard_range_rows() ff83295 Improve building listings from shards 37dafa0 Fix probe tests in the gate 8dc9bc1 Use sets instead of dicts when we only care about the keys 9530ab2 Pre-storage-policy-index tests shouldn't have shard_range tables 99023ef Maintain manage-shard-ranges docs in its module cc565db Remove exclude_states from get_shard_ranges 83732a6 Use assertRaises more 4c559fc Container sharding doc fixes and clarifications c35285f Use correct policy when faking misplaced objects in probe test 5c5b08d Add container sharding documentation 2641814 Add sharder daemon, manage_shard_ranges tool and probe tests 4a3efe6 Redirect object updates to shard containers e940bc6 Enable proxy to build listings from shards 723eac9 Add shard range support to container server 14af38a Add support for sharding in ContainerBroker a962340 Add ShardRange class 9d742b8 Refactoring, test infrastructure changes and cleanup 0d0c9b9 s3api: simplify BaseAclHandler.request_with 5257ed0 Fix indentation in overview_backin_store.rst cdce4d9 fix s3api docstring 2c7768a Small cleanup on s3api 04ee763 Simplify the code in s3api c94acbc updated .gitreview 636b922 Import swift3 into swift repo as s3api middleware c4751d0 Make reconstructor go faster with --override-devices c28004d Multiprocess object replicator 8085807 py3: port gatekeeper 9b90498 Trivial: Update pypi url to new url 87dad29 added another golang client to associated projects 708b24a Deprecate auth_uri option 6a428c4 updated associated projects page 2307886 Truncate error logs to prevent log handler runs out of the buffer. 817ca07 py3: port cli recon tests 83ca4f1 Handful of formpost cleanups f1bbb5f Clean up SegmentError messages 5b68eb5 swift-(account|container)-info: tolerate LockTimeouts c42f15f Add ability to run specific tests in py35 tox environment. b0b3ed0 Improve check for O_TMPFILE support in unit tests 77bfe7a swob.Match: add __repr__ 9a865f8 update requirements to support matching global exclusions 8944e9d Support unicode characters in tempauth user names. 5d20411 Don't log tracebacks on ECONNRESET, ENETUNREACH, BadStatusLine 42adbe5 Respect X-Backend-Etag-Is-At headers from left of SLO baa4fa5 swob.Match: Optional whitespace is optional a027f2c Follow the new PTI for document build 9b4c978 Use http code constant instead of int 91e5cba Use http code constant instead of int ecf8ae5 Remove object replicator's lockup detector/mitigator. 3bc267d fix a typo in documentation 8d7e245 Change object_audit_location_generator() to yield for a single policy. 9f4910f Add round_robin_iter function to common/utils 7e77bf3 Refactor tempurl functional test's try-finally 2c4c23d Encapsulate some general utility in utils bb3553d Use state when you have it de38b72 Don't double-filter replication jobs 22b9a4a Fix tests using O_TMPFILE e84ed57 py3: port cli form_signature and unit tests 1b546f6 Capture probe test logs e5ce83b Update links in README bac79f2 Unify expirer unit test task queue e464049 py3: have Timestamp accept bytestrings, better validate input 335899f Functional tests for staticweb middleware b640631 Apply remote metadata in _handle_sync_response ca21d71 Add swift probe tests to gate 63e6c75 Temporarily disable flaky test. b08c70d Support -d <devs> and -p <partitions> in DB replicators. afb6cb5 Try to avoid leaving (killed) long-running rsyncs in the process table 57b632f Fix object-server to not 400 all expirer DELETEs 9eeacbf Fix new doc warnings resulting from Sphinx 1.7.1 fdaf052 Make test_greater_with_offset not fail on py36 8b8a2a3 Tolerate 404s during setUp/tearDown in func tests 10943b8 Imported Translations from Zanata 36c4297 py3: Port more CLI tools 624b531 py3: port common/wsgi.py d09d60d Update release notes for stable/queens 7eb9852 Don't log tracebacks when we know where they're going to end 1fadffe Split expirer methods and parametrize task account 9e5f434 Kill rsync coros when lockup detector tries to kill the process b3f1558 Fix expirer's invalid task object names in unit tests 5cb0869 py3: port common/memcached.py 4b19ac7 py3: port common/storage_policy.py 25540a4 Tighten up assertions around expirer's concurrency 5017864 Fix the incorrect reference links 532ac9e Ensure reverting test env if the env is temporarily changed 58f5d89 Remove confusing assertion from expirer's unit test 6060af8 Add more tests around ObjectExpirer.round_robin_order 3036353 Refactor expirer's task round robin implementation c6169cd Fix a typo in swift/common/swob.py 47fed6f Add handoffs-only mode to DB replicators. 2bfd9c6 Make DB replicators ignore non-partition directories 8539b93 Small cleanup in DB replicator's error handling 5730618 Improved usage of args in .functests 8dc4012 add keystonemiddleware to extras area 642f799 py3: port common/ring/ and common/utils.py 1dddcb5 probetests: Close our handle to subprocess' stdouts f0f3de3 fix typos in swift 54918ad Fix grammar error 7537154 Imported Translations from Zanata a3d2aab Fix typos in swift 1f4ebbc kill orphans during probe test setup c9410c7 Move eventlet patch before call to loadapp 0a99343 Update install-guide URLs to point to stable/queens 6b9098e Imported Translations from Zanata bfe52a2 Quarantine DB without *_stat row 5b30c1f Fix flakey test_check_delete_headers_sets_delete_at d9eec63 Zuul: Remove project name 98d1859 Cleanup for iterators in SegmentedIterable 3991055 Solve the zombie process problem of Auditor 8140b7e Fix inconsistency of account info in expirer's unit tests 38ada71 Make statsd errors correspond to 5xx only 611b28f Add manpage for swift-object-relinker dd9bc82 Fix suffix-byte-range responses for zero-byte replicated objects. 78c7d4c Using assertIsNone() instead of assertEqual(None) f95befb Optionally drop common prefixes in swift-*-info output 33714c2 Add full working example of sharing a container with another user 87cc550 Log deprecation warnings for account_white/blacklist 9d12575 swift-recon-cron: do not get confused by files in /srv/node Diffstat (except docs and test files) ------------------------------------- .mailmap | 3 + .zuul.yaml | 47 +- AUTHORS | 17 + CHANGELOG | 76 +- README.rst | 4 +- bin/swift-account-info | 6 +- bin/swift-account-replicator | 13 +- bin/swift-container-info | 6 +- bin/swift-container-replicator | 13 +- bin/swift-container-sharder | 33 + bin/swift-object-info | 11 + bin/swift-object-relinker | 6 +- bin/swift-recon-cron | 2 + bindep.txt | 4 + .../finalize-installation-ubuntu-debian.rst | 2 +- .../install/storage-install-ubuntu-debian.rst | 6 +- etc/account-server.conf-sample | 19 + etc/container-server.conf-sample | 137 + etc/keymaster.conf-sample | 2 +- etc/object-server.conf-sample | 11 +- etc/proxy-server.conf-sample | 152 +- .../install_dependencies.yaml | 50 + playbooks/saio_single_node_setup/make_rings.yaml | 29 + playbooks/saio_single_node_setup/setup_saio.yaml | 174 + .../saio_single_node_setup/templates/rc.local.j2 | 8 + .../notes/2_18_0_release-3acf63cfe2475c65.yaml | 85 + releasenotes/source/index.rst | 2 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 175 +- .../source/locale/ja/LC_MESSAGES/releasenotes.po | 375 +- .../locale/ko_KR/LC_MESSAGES/releasenotes.po | 75 + releasenotes/source/queens.rst | 6 + requirements.txt | 6 +- setup.cfg | 14 +- swift/account/backend.py | 11 +- swift/cli/form_signature.py | 11 +- swift/cli/info.py | 76 +- swift/cli/manage_shard_ranges.py | 515 +++ swift/cli/recon.py | 4 +- swift/cli/ringbuilder.py | 2 +- swift/cli/shard-info.py | 195 + swift/common/daemon.py | 13 +- swift/common/db.py | 285 +- swift/common/db_replicator.py | 291 +- swift/common/direct_client.py | 150 +- swift/common/linkat.py | 7 + swift/common/manager.py | 22 +- swift/common/memcached.py | 82 +- swift/common/middleware/bulk.py | 2 +- swift/common/middleware/formpost.py | 18 +- swift/common/middleware/gatekeeper.py | 2 +- swift/common/middleware/ratelimit.py | 10 + swift/common/middleware/s3api/__init__.py | 0 swift/common/middleware/s3api/acl_handlers.py | 463 ++ swift/common/middleware/s3api/acl_utils.py | 95 + .../middleware/s3api/controllers/__init__.py | 52 + swift/common/middleware/s3api/controllers/acl.py | 130 + swift/common/middleware/s3api/controllers/base.py | 100 + .../common/middleware/s3api/controllers/bucket.py | 251 ++ .../middleware/s3api/controllers/location.py | 42 + .../common/middleware/s3api/controllers/logging.py | 54 + .../middleware/s3api/controllers/multi_delete.py | 126 + .../middleware/s3api/controllers/multi_upload.py | 671 +++ swift/common/middleware/s3api/controllers/obj.py | 150 + .../common/middleware/s3api/controllers/s3_acl.py | 67 + .../common/middleware/s3api/controllers/service.py | 68 + .../middleware/s3api/controllers/versioning.py | 53 + swift/common/middleware/s3api/etree.py | 146 + swift/common/middleware/s3api/exception.py | 36 + swift/common/middleware/s3api/s3api.py | 273 ++ swift/common/middleware/s3api/s3request.py | 1402 ++++++ swift/common/middleware/s3api/s3response.py | 684 +++ swift/common/middleware/s3api/s3token.py | 324 ++ .../s3api/schema/access_control_policy.rng | 16 + .../s3api/schema/bucket_logging_status.rng | 25 + swift/common/middleware/s3api/schema/common.rng | 66 + .../s3api/schema/complete_multipart_upload.rng | 19 + .../schema/complete_multipart_upload_result.rng | 19 + .../middleware/s3api/schema/copy_object_result.rng | 13 + .../middleware/s3api/schema/copy_part_result.rng | 13 + .../s3api/schema/create_bucket_configuration.rng | 11 + swift/common/middleware/s3api/schema/delete.rng | 28 + .../middleware/s3api/schema/delete_result.rng | 47 + swift/common/middleware/s3api/schema/error.rng | 30 + .../schema/initiate_multipart_upload_result.rng | 16 + .../s3api/schema/lifecycle_configuration.rng | 56 + .../s3api/schema/list_all_my_buckets_result.rng | 23 + .../middleware/s3api/schema/list_bucket_result.rng | 93 + .../s3api/schema/list_multipart_uploads_result.rng | 73 + .../middleware/s3api/schema/list_parts_result.rng | 59 + .../s3api/schema/list_versions_result.rng | 104 + .../s3api/schema/location_constraint.rng | 8 + .../s3api/schema/versioning_configuration.rng | 25 + swift/common/middleware/s3api/subresource.py | 563 +++ swift/common/middleware/s3api/utils.py | 189 + swift/common/middleware/slo.py | 38 +- swift/common/middleware/symlink.py | 6 +- swift/common/middleware/tempauth.py | 9 +- swift/common/middleware/versioned_writes.py | 52 +- swift/common/request_helpers.py | 201 +- swift/common/ring/builder.py | 20 +- swift/common/ring/composite_builder.py | 6 +- swift/common/ring/ring.py | 14 +- swift/common/splice.py | 7 +- swift/common/storage_policy.py | 20 +- swift/common/swob.py | 33 +- swift/common/utils.py | 1013 ++++- swift/common/wsgi.py | 175 +- swift/container/backend.py | 1336 +++++- swift/container/replicator.py | 130 +- swift/container/server.py | 225 +- swift/container/sharder.py | 1568 +++++++ swift/container/updater.py | 9 +- swift/locale/en_GB/LC_MESSAGES/swift.po | 24 +- swift/locale/es/LC_MESSAGES/swift.po | 21 +- swift/locale/ja/LC_MESSAGES/swift.po | 112 +- swift/locale/ko_KR/LC_MESSAGES/swift.po | 6 +- swift/locale/pt_BR/LC_MESSAGES/swift.po | 6 +- swift/locale/tr_TR/LC_MESSAGES/swift.po | 6 +- swift/locale/zh_CN/LC_MESSAGES/swift.po | 6 +- swift/locale/zh_TW/LC_MESSAGES/swift.po | 6 +- swift/obj/auditor.py | 38 +- swift/obj/diskfile.py | 159 +- swift/obj/expirer.py | 320 +- swift/obj/reconstructor.py | 88 +- swift/obj/replicator.py | 538 ++- swift/obj/server.py | 83 +- swift/obj/updater.py | 165 +- swift/proxy/controllers/base.py | 97 +- swift/proxy/controllers/container.py | 125 +- swift/proxy/controllers/obj.py | 45 +- swift/proxy/server.py | 5 +- test-requirements.txt | 8 +- test/__init__.py | 27 + test/functional/__init__.py | 131 +- test/functional/s3api/__init__.py | 61 + test/functional/s3api/s3_test_client.py | 139 + test/functional/s3api/test_acl.py | 156 + test/functional/s3api/test_bucket.py | 487 +++ test/functional/s3api/test_multi_delete.py | 248 ++ test/functional/s3api/test_multi_upload.py | 849 ++++ test/functional/s3api/test_object.py | 873 ++++ test/functional/s3api/test_presigned.py | 237 + test/functional/s3api/test_service.py | 100 + test/functional/s3api/utils.py | 31 + test/functional/swift_test_client.py | 13 +- test/functional/test_staticweb.py | 400 ++ test/functional/test_tempurl.py | 6 +- test/functional/test_versioned_writes.py | 123 +- test/probe/brain.py | 6 +- test/probe/common.py | 116 +- test/probe/test_db_replicator.py | 13 +- test/probe/test_object_expirer.py | 36 +- test/probe/test_sharder.py | 2034 +++++++++ test/sample.conf | 5 + test/unit/__init__.py | 76 +- test/unit/account/test_server.py | 4 +- test/unit/cli/test_form_signature.py | 31 +- test/unit/cli/test_info.py | 170 +- test/unit/cli/test_manage_shard_ranges.py | 362 ++ test/unit/cli/test_recon.py | 40 +- test/unit/cli/test_relinker.py | 4 +- test/unit/cli/test_ringbuilder.py | 8 +- test/unit/common/middleware/helpers.py | 9 +- test/unit/common/middleware/s3api/__init__.py | 163 + test/unit/common/middleware/s3api/exceptions.py | 18 + test/unit/common/middleware/s3api/helpers.py | 185 + test/unit/common/middleware/s3api/test_acl.py | 230 + .../common/middleware/s3api/test_acl_handlers.py | 42 + .../unit/common/middleware/s3api/test_acl_utils.py | 49 + test/unit/common/middleware/s3api/test_bucket.py | 755 ++++ test/unit/common/middleware/s3api/test_cfg.py | 44 + test/unit/common/middleware/s3api/test_etree.py | 73 + test/unit/common/middleware/s3api/test_helpers.py | 69 + test/unit/common/middleware/s3api/test_location.py | 51 + test/unit/common/middleware/s3api/test_logging.py | 66 + .../common/middleware/s3api/test_multi_delete.py | 284 ++ .../common/middleware/s3api/test_multi_upload.py | 1742 ++++++++ test/unit/common/middleware/s3api/test_obj.py | 1010 +++++ test/unit/common/middleware/s3api/test_s3_acl.py | 540 +++ test/unit/common/middleware/s3api/test_s3api.py | 1049 +++++ .../unit/common/middleware/s3api/test_s3request.py | 765 ++++ .../common/middleware/s3api/test_s3response.py | 80 + test/unit/common/middleware/s3api/test_s3token.py | 821 ++++ test/unit/common/middleware/s3api/test_service.py | 235 + .../common/middleware/s3api/test_subresource.py | 367 ++ test/unit/common/middleware/s3api/test_utils.py | 133 + .../common/middleware/s3api/test_versioning.py | 56 + test/unit/common/middleware/test_gatekeeper.py | 2 +- test/unit/common/middleware/test_slo.py | 217 +- test/unit/common/middleware/test_tempauth.py | 45 +- .../common/middleware/test_versioned_writes.py | 24 + test/unit/common/missing_container_info.db | Bin 0 -> 17408 bytes test/unit/common/ring/test_builder.py | 15 +- test/unit/common/ring/test_composite_builder.py | 89 +- test/unit/common/ring/test_ring.py | 22 +- test/unit/common/ring/test_utils.py | 3 + test/unit/common/test_constraints.py | 18 +- test/unit/common/test_daemon.py | 8 + test/unit/common/test_db.py | 393 +- test/unit/common/test_db_replicator.py | 615 ++- test/unit/common/test_direct_client.py | 70 + test/unit/common/test_linkat.py | 6 +- test/unit/common/test_memcached.py | 158 +- test/unit/common/test_storage_policy.py | 9 +- test/unit/common/test_swob.py | 16 + test/unit/common/test_utils.py | 1586 ++++++- test/unit/common/test_wsgi.py | 336 +- test/unit/container/test_backend.py | 2532 ++++++++++- test/unit/container/test_replicator.py | 1112 ++++- test/unit/container/test_server.py | 1268 +++++- test/unit/container/test_sharder.py | 4580 ++++++++++++++++++++ test/unit/container/test_updater.py | 26 +- test/unit/helpers.py | 2 +- test/unit/obj/test_auditor.py | 42 +- test/unit/obj/test_diskfile.py | 116 +- test/unit/obj/test_expirer.py | 582 ++- test/unit/obj/test_reconstructor.py | 77 +- test/unit/obj/test_replicator.py | 713 ++- test/unit/obj/test_server.py | 303 +- test/unit/obj/test_ssync_sender.py | 19 +- test/unit/obj/test_updater.py | 466 +- test/unit/proxy/controllers/test_base.py | 154 +- test/unit/proxy/controllers/test_container.py | 1024 ++++- test/unit/proxy/test_server.py | 289 +- tools/test-setup.sh | 2 +- tox.ini | 42 +- 295 files changed, 62272 insertions(+), 2672 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 1001be5..c115113 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5 +5 @@ -dnspython>=1.14.0 # http://www.dnspython.org/LICENSE +dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE @@ -9 +9,3 @@ netifaces>=0.5,!=0.10.0,!=0.10.1 -pastedeploy>=1.3.3 +PasteDeploy>=1.3.3 +lxml +requests>=2.14.2 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 968571a..5f515bd 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -11,3 +10,0 @@ nosehtmloutput>=0.0.3 # Apache-2.0 -sphinx>=1.6.2 # BSD -openstackdocstheme>=1.11.0 # Apache-2.0 -os-api-ref>=1.0.0 # Apache-2.0 @@ -18,0 +16,5 @@ reno>=1.8.0 # Apache-2.0 +python-openstackclient +boto +requests-mock>=1.2.0 # Apache-2.0 +fixtures>=3.0.0 # Apache-2.0/BSD +keystonemiddleware>=4.17.0 # Apache-2.0