[release-announce] swift 2.18.0 (rocky)
no-reply at openstack.org
no-reply at openstack.org
Wed May 30 10:24:54 UTC 2018
We are pumped to announce the release of:
swift 2.18.0: OpenStack Object Storage
This release is part of the rocky release series.
Download the package from:
https://tarballs.openstack.org/swift/
For more details, please see below.
2.18.0
^^^^^^
New Features
* Added container sharding, an operator controlled feature that may
be used to shard very large container databases into a number of
smaller shard containers. This mitigates the issues with one large
DB by distributing the data across multiple smaller databases
throughout the cluster. Please read the full overview at https://do
cs.openstack.org/swift/latest/overview_container_sharding.html
* Provide an S3 API compatibility layer. The external "swift3"
project has been imported into Swift's codebase as the "s3api"
middleware.
* Added "emergency mode" hooks in the account and container
replicators. These options may be used to prioritize moving handoff
partitions to primary locations more quickly. This helps when adding
capacity to a ring.
* Added "-d <devs>" and "-p <partitions>" command line options.
* Added a handoffs-only mode.
* Add a multiprocess mode to the object replicator. Setting the
"replicator_workers" setting to a positive value N will result in
the replicator using up to N worker processes to perform replication
tasks. At most one worker per disk will be spawned.
Worker process logs will have a bit of information prepended so
operators can tell which messages came from which worker. The prefix
is "[worker M/N pid=P] ", where M is the worker's index, N is the
total number of workers, and P is the process ID. Every message from
the replicator's logger will have the prefix
* The object reconstructor will now fork all available worker
processes when operating on a subset of local devices.
* Add support for PROXY protocol v1 to the proxy server. This allows
the Swift proxy server to log accurate client IP addresses when
there is a proxy or SSL-terminator between the client and the Swift
proxy server. Example servers supporting this PROXY protocol
include stunnel, haproxy, hitch, and varnish. See the sample proxy
server config file for the appropriate config setting to enable or
disable this functionality.
* In the ratelimit middleware, account whitelist and blacklist
settings have been deprecated and may be removed in a future
release. When found, a deprecation message will be logged. Instead
of these config file values, set X-Account-Sysmeta- Global-Write-
Ratelimit:WHITELIST and X-Account-Sysmeta-Global- Write-
Ratelimit:BLACKLIST on the particular accounts that need to be
whitelisted or blacklisted. System metadata cannot be added or
modified by standard clients. Use the internal client to set
sysmeta.
* Add a "--drop-prefixes" flag to swift-account-info, swift-
container- info, and swift-object-info. This makes the output
between the three more consistent.
* statsd error messages correspond to 5xx responses only. This makes
monitoring more useful because actual errors (5xx) will not be
hidden by common user requests (4xx). Previously, some 4xx responses
would be included in timing information in the statsd error
messages.
* Truncate error logs to prevent log handler from running out of
buffer.
* Updated requirements.txt to match global exclusions and
formatting.
* tempauth user names now support unicode characters.
* Various other minor bug fixes and improvements.
Changes in swift 2.17.0..2.18.0
-------------------------------
fbb3b1f authors/changelog update for 2.18.0 release
0a964f8 add s3_acl option to s3api /info
36dbd38 Add s3api headers to allowed_headers by default
773b633 Change default sharding threshold to 1,000,000 objects
c341c2c fix typos in comment
4189a11 catch lock fail exception in container updater
71b39e4 Remove un-used stub
fbb8d7e Clarify that archive location headers should be URL-encoded
d52a527 Clean out Python 2.6 leftovers from splice.py
a6f7600 Remove our reimplemented logging.NullHandler
693d9a6 Always pass capitalize_response_headers=False to eventlet.wsgi.server()
661838d Add support for PROXY protocol v1 (only)
0aad950 Fix SLO delete for accounts with non-ASCII names.
e4045fb Add unit tests for replicator sync_shard_ranges
3d3393c Fix typo
6714af8 Use maybe_get more
2e321d9 Add a helper context for optional connection args
4af57db Let make_db_file_path accept epoch=None
f68dd3b Move handling of missing table outside of do_query
8486397 Clean up logging interpolation
8c386ff Make the decision between primary/handoff sets more obvious
0e3e7b9 Fix versioned writes error with url-encoded object name
f237c58 Bump up timeout for swift-probetests-centos-7
37ee89e Avoid premature shrinking in sharder probe test
82314bd Test rsync error log truncation
0ee2a1d Simplify ContainerBroker._get_shard_range_rows()
ff83295 Improve building listings from shards
37dafa0 Fix probe tests in the gate
8dc9bc1 Use sets instead of dicts when we only care about the keys
9530ab2 Pre-storage-policy-index tests shouldn't have shard_range tables
99023ef Maintain manage-shard-ranges docs in its module
cc565db Remove exclude_states from get_shard_ranges
83732a6 Use assertRaises more
4c559fc Container sharding doc fixes and clarifications
c35285f Use correct policy when faking misplaced objects in probe test
5c5b08d Add container sharding documentation
2641814 Add sharder daemon, manage_shard_ranges tool and probe tests
4a3efe6 Redirect object updates to shard containers
e940bc6 Enable proxy to build listings from shards
723eac9 Add shard range support to container server
14af38a Add support for sharding in ContainerBroker
a962340 Add ShardRange class
9d742b8 Refactoring, test infrastructure changes and cleanup
0d0c9b9 s3api: simplify BaseAclHandler.request_with
5257ed0 Fix indentation in overview_backin_store.rst
cdce4d9 fix s3api docstring
2c7768a Small cleanup on s3api
04ee763 Simplify the code in s3api
c94acbc updated .gitreview
636b922 Import swift3 into swift repo as s3api middleware
c4751d0 Make reconstructor go faster with --override-devices
c28004d Multiprocess object replicator
8085807 py3: port gatekeeper
9b90498 Trivial: Update pypi url to new url
87dad29 added another golang client to associated projects
708b24a Deprecate auth_uri option
6a428c4 updated associated projects page
2307886 Truncate error logs to prevent log handler runs out of the buffer.
817ca07 py3: port cli recon tests
83ca4f1 Handful of formpost cleanups
f1bbb5f Clean up SegmentError messages
5b68eb5 swift-(account|container)-info: tolerate LockTimeouts
c42f15f Add ability to run specific tests in py35 tox environment.
b0b3ed0 Improve check for O_TMPFILE support in unit tests
77bfe7a swob.Match: add __repr__
9a865f8 update requirements to support matching global exclusions
8944e9d Support unicode characters in tempauth user names.
5d20411 Don't log tracebacks on ECONNRESET, ENETUNREACH, BadStatusLine
42adbe5 Respect X-Backend-Etag-Is-At headers from left of SLO
baa4fa5 swob.Match: Optional whitespace is optional
a027f2c Follow the new PTI for document build
9b4c978 Use http code constant instead of int
91e5cba Use http code constant instead of int
ecf8ae5 Remove object replicator's lockup detector/mitigator.
3bc267d fix a typo in documentation
8d7e245 Change object_audit_location_generator() to yield for a single policy.
9f4910f Add round_robin_iter function to common/utils
7e77bf3 Refactor tempurl functional test's try-finally
2c4c23d Encapsulate some general utility in utils
bb3553d Use state when you have it
de38b72 Don't double-filter replication jobs
22b9a4a Fix tests using O_TMPFILE
e84ed57 py3: port cli form_signature and unit tests
1b546f6 Capture probe test logs
e5ce83b Update links in README
bac79f2 Unify expirer unit test task queue
e464049 py3: have Timestamp accept bytestrings, better validate input
335899f Functional tests for staticweb middleware
b640631 Apply remote metadata in _handle_sync_response
ca21d71 Add swift probe tests to gate
63e6c75 Temporarily disable flaky test.
b08c70d Support -d <devs> and -p <partitions> in DB replicators.
afb6cb5 Try to avoid leaving (killed) long-running rsyncs in the process table
57b632f Fix object-server to not 400 all expirer DELETEs
9eeacbf Fix new doc warnings resulting from Sphinx 1.7.1
fdaf052 Make test_greater_with_offset not fail on py36
8b8a2a3 Tolerate 404s during setUp/tearDown in func tests
10943b8 Imported Translations from Zanata
36c4297 py3: Port more CLI tools
624b531 py3: port common/wsgi.py
d09d60d Update release notes for stable/queens
7eb9852 Don't log tracebacks when we know where they're going to end
1fadffe Split expirer methods and parametrize task account
9e5f434 Kill rsync coros when lockup detector tries to kill the process
b3f1558 Fix expirer's invalid task object names in unit tests
5cb0869 py3: port common/memcached.py
4b19ac7 py3: port common/storage_policy.py
25540a4 Tighten up assertions around expirer's concurrency
5017864 Fix the incorrect reference links
532ac9e Ensure reverting test env if the env is temporarily changed
58f5d89 Remove confusing assertion from expirer's unit test
6060af8 Add more tests around ObjectExpirer.round_robin_order
3036353 Refactor expirer's task round robin implementation
c6169cd Fix a typo in swift/common/swob.py
47fed6f Add handoffs-only mode to DB replicators.
2bfd9c6 Make DB replicators ignore non-partition directories
8539b93 Small cleanup in DB replicator's error handling
5730618 Improved usage of args in .functests
8dc4012 add keystonemiddleware to extras area
642f799 py3: port common/ring/ and common/utils.py
1dddcb5 probetests: Close our handle to subprocess' stdouts
f0f3de3 fix typos in swift
54918ad Fix grammar error
7537154 Imported Translations from Zanata
a3d2aab Fix typos in swift
1f4ebbc kill orphans during probe test setup
c9410c7 Move eventlet patch before call to loadapp
0a99343 Update install-guide URLs to point to stable/queens
6b9098e Imported Translations from Zanata
bfe52a2 Quarantine DB without *_stat row
5b30c1f Fix flakey test_check_delete_headers_sets_delete_at
d9eec63 Zuul: Remove project name
98d1859 Cleanup for iterators in SegmentedIterable
3991055 Solve the zombie process problem of Auditor
8140b7e Fix inconsistency of account info in expirer's unit tests
38ada71 Make statsd errors correspond to 5xx only
611b28f Add manpage for swift-object-relinker
dd9bc82 Fix suffix-byte-range responses for zero-byte replicated objects.
78c7d4c Using assertIsNone() instead of assertEqual(None)
f95befb Optionally drop common prefixes in swift-*-info output
33714c2 Add full working example of sharing a container with another user
87cc550 Log deprecation warnings for account_white/blacklist
9d12575 swift-recon-cron: do not get confused by files in /srv/node
Diffstat (except docs and test files)
-------------------------------------
.mailmap | 3 +
.zuul.yaml | 47 +-
AUTHORS | 17 +
CHANGELOG | 76 +-
README.rst | 4 +-
bin/swift-account-info | 6 +-
bin/swift-account-replicator | 13 +-
bin/swift-container-info | 6 +-
bin/swift-container-replicator | 13 +-
bin/swift-container-sharder | 33 +
bin/swift-object-info | 11 +
bin/swift-object-relinker | 6 +-
bin/swift-recon-cron | 2 +
bindep.txt | 4 +
.../finalize-installation-ubuntu-debian.rst | 2 +-
.../install/storage-install-ubuntu-debian.rst | 6 +-
etc/account-server.conf-sample | 19 +
etc/container-server.conf-sample | 137 +
etc/keymaster.conf-sample | 2 +-
etc/object-server.conf-sample | 11 +-
etc/proxy-server.conf-sample | 152 +-
.../install_dependencies.yaml | 50 +
playbooks/saio_single_node_setup/make_rings.yaml | 29 +
playbooks/saio_single_node_setup/setup_saio.yaml | 174 +
.../saio_single_node_setup/templates/rc.local.j2 | 8 +
.../notes/2_18_0_release-3acf63cfe2475c65.yaml | 85 +
releasenotes/source/index.rst | 2 +
.../locale/en_GB/LC_MESSAGES/releasenotes.po | 175 +-
.../source/locale/ja/LC_MESSAGES/releasenotes.po | 375 +-
.../locale/ko_KR/LC_MESSAGES/releasenotes.po | 75 +
releasenotes/source/queens.rst | 6 +
requirements.txt | 6 +-
setup.cfg | 14 +-
swift/account/backend.py | 11 +-
swift/cli/form_signature.py | 11 +-
swift/cli/info.py | 76 +-
swift/cli/manage_shard_ranges.py | 515 +++
swift/cli/recon.py | 4 +-
swift/cli/ringbuilder.py | 2 +-
swift/cli/shard-info.py | 195 +
swift/common/daemon.py | 13 +-
swift/common/db.py | 285 +-
swift/common/db_replicator.py | 291 +-
swift/common/direct_client.py | 150 +-
swift/common/linkat.py | 7 +
swift/common/manager.py | 22 +-
swift/common/memcached.py | 82 +-
swift/common/middleware/bulk.py | 2 +-
swift/common/middleware/formpost.py | 18 +-
swift/common/middleware/gatekeeper.py | 2 +-
swift/common/middleware/ratelimit.py | 10 +
swift/common/middleware/s3api/__init__.py | 0
swift/common/middleware/s3api/acl_handlers.py | 463 ++
swift/common/middleware/s3api/acl_utils.py | 95 +
.../middleware/s3api/controllers/__init__.py | 52 +
swift/common/middleware/s3api/controllers/acl.py | 130 +
swift/common/middleware/s3api/controllers/base.py | 100 +
.../common/middleware/s3api/controllers/bucket.py | 251 ++
.../middleware/s3api/controllers/location.py | 42 +
.../common/middleware/s3api/controllers/logging.py | 54 +
.../middleware/s3api/controllers/multi_delete.py | 126 +
.../middleware/s3api/controllers/multi_upload.py | 671 +++
swift/common/middleware/s3api/controllers/obj.py | 150 +
.../common/middleware/s3api/controllers/s3_acl.py | 67 +
.../common/middleware/s3api/controllers/service.py | 68 +
.../middleware/s3api/controllers/versioning.py | 53 +
swift/common/middleware/s3api/etree.py | 146 +
swift/common/middleware/s3api/exception.py | 36 +
swift/common/middleware/s3api/s3api.py | 273 ++
swift/common/middleware/s3api/s3request.py | 1402 ++++++
swift/common/middleware/s3api/s3response.py | 684 +++
swift/common/middleware/s3api/s3token.py | 324 ++
.../s3api/schema/access_control_policy.rng | 16 +
.../s3api/schema/bucket_logging_status.rng | 25 +
swift/common/middleware/s3api/schema/common.rng | 66 +
.../s3api/schema/complete_multipart_upload.rng | 19 +
.../schema/complete_multipart_upload_result.rng | 19 +
.../middleware/s3api/schema/copy_object_result.rng | 13 +
.../middleware/s3api/schema/copy_part_result.rng | 13 +
.../s3api/schema/create_bucket_configuration.rng | 11 +
swift/common/middleware/s3api/schema/delete.rng | 28 +
.../middleware/s3api/schema/delete_result.rng | 47 +
swift/common/middleware/s3api/schema/error.rng | 30 +
.../schema/initiate_multipart_upload_result.rng | 16 +
.../s3api/schema/lifecycle_configuration.rng | 56 +
.../s3api/schema/list_all_my_buckets_result.rng | 23 +
.../middleware/s3api/schema/list_bucket_result.rng | 93 +
.../s3api/schema/list_multipart_uploads_result.rng | 73 +
.../middleware/s3api/schema/list_parts_result.rng | 59 +
.../s3api/schema/list_versions_result.rng | 104 +
.../s3api/schema/location_constraint.rng | 8 +
.../s3api/schema/versioning_configuration.rng | 25 +
swift/common/middleware/s3api/subresource.py | 563 +++
swift/common/middleware/s3api/utils.py | 189 +
swift/common/middleware/slo.py | 38 +-
swift/common/middleware/symlink.py | 6 +-
swift/common/middleware/tempauth.py | 9 +-
swift/common/middleware/versioned_writes.py | 52 +-
swift/common/request_helpers.py | 201 +-
swift/common/ring/builder.py | 20 +-
swift/common/ring/composite_builder.py | 6 +-
swift/common/ring/ring.py | 14 +-
swift/common/splice.py | 7 +-
swift/common/storage_policy.py | 20 +-
swift/common/swob.py | 33 +-
swift/common/utils.py | 1013 ++++-
swift/common/wsgi.py | 175 +-
swift/container/backend.py | 1336 +++++-
swift/container/replicator.py | 130 +-
swift/container/server.py | 225 +-
swift/container/sharder.py | 1568 +++++++
swift/container/updater.py | 9 +-
swift/locale/en_GB/LC_MESSAGES/swift.po | 24 +-
swift/locale/es/LC_MESSAGES/swift.po | 21 +-
swift/locale/ja/LC_MESSAGES/swift.po | 112 +-
swift/locale/ko_KR/LC_MESSAGES/swift.po | 6 +-
swift/locale/pt_BR/LC_MESSAGES/swift.po | 6 +-
swift/locale/tr_TR/LC_MESSAGES/swift.po | 6 +-
swift/locale/zh_CN/LC_MESSAGES/swift.po | 6 +-
swift/locale/zh_TW/LC_MESSAGES/swift.po | 6 +-
swift/obj/auditor.py | 38 +-
swift/obj/diskfile.py | 159 +-
swift/obj/expirer.py | 320 +-
swift/obj/reconstructor.py | 88 +-
swift/obj/replicator.py | 538 ++-
swift/obj/server.py | 83 +-
swift/obj/updater.py | 165 +-
swift/proxy/controllers/base.py | 97 +-
swift/proxy/controllers/container.py | 125 +-
swift/proxy/controllers/obj.py | 45 +-
swift/proxy/server.py | 5 +-
test-requirements.txt | 8 +-
test/__init__.py | 27 +
test/functional/__init__.py | 131 +-
test/functional/s3api/__init__.py | 61 +
test/functional/s3api/s3_test_client.py | 139 +
test/functional/s3api/test_acl.py | 156 +
test/functional/s3api/test_bucket.py | 487 +++
test/functional/s3api/test_multi_delete.py | 248 ++
test/functional/s3api/test_multi_upload.py | 849 ++++
test/functional/s3api/test_object.py | 873 ++++
test/functional/s3api/test_presigned.py | 237 +
test/functional/s3api/test_service.py | 100 +
test/functional/s3api/utils.py | 31 +
test/functional/swift_test_client.py | 13 +-
test/functional/test_staticweb.py | 400 ++
test/functional/test_tempurl.py | 6 +-
test/functional/test_versioned_writes.py | 123 +-
test/probe/brain.py | 6 +-
test/probe/common.py | 116 +-
test/probe/test_db_replicator.py | 13 +-
test/probe/test_object_expirer.py | 36 +-
test/probe/test_sharder.py | 2034 +++++++++
test/sample.conf | 5 +
test/unit/__init__.py | 76 +-
test/unit/account/test_server.py | 4 +-
test/unit/cli/test_form_signature.py | 31 +-
test/unit/cli/test_info.py | 170 +-
test/unit/cli/test_manage_shard_ranges.py | 362 ++
test/unit/cli/test_recon.py | 40 +-
test/unit/cli/test_relinker.py | 4 +-
test/unit/cli/test_ringbuilder.py | 8 +-
test/unit/common/middleware/helpers.py | 9 +-
test/unit/common/middleware/s3api/__init__.py | 163 +
test/unit/common/middleware/s3api/exceptions.py | 18 +
test/unit/common/middleware/s3api/helpers.py | 185 +
test/unit/common/middleware/s3api/test_acl.py | 230 +
.../common/middleware/s3api/test_acl_handlers.py | 42 +
.../unit/common/middleware/s3api/test_acl_utils.py | 49 +
test/unit/common/middleware/s3api/test_bucket.py | 755 ++++
test/unit/common/middleware/s3api/test_cfg.py | 44 +
test/unit/common/middleware/s3api/test_etree.py | 73 +
test/unit/common/middleware/s3api/test_helpers.py | 69 +
test/unit/common/middleware/s3api/test_location.py | 51 +
test/unit/common/middleware/s3api/test_logging.py | 66 +
.../common/middleware/s3api/test_multi_delete.py | 284 ++
.../common/middleware/s3api/test_multi_upload.py | 1742 ++++++++
test/unit/common/middleware/s3api/test_obj.py | 1010 +++++
test/unit/common/middleware/s3api/test_s3_acl.py | 540 +++
test/unit/common/middleware/s3api/test_s3api.py | 1049 +++++
.../unit/common/middleware/s3api/test_s3request.py | 765 ++++
.../common/middleware/s3api/test_s3response.py | 80 +
test/unit/common/middleware/s3api/test_s3token.py | 821 ++++
test/unit/common/middleware/s3api/test_service.py | 235 +
.../common/middleware/s3api/test_subresource.py | 367 ++
test/unit/common/middleware/s3api/test_utils.py | 133 +
.../common/middleware/s3api/test_versioning.py | 56 +
test/unit/common/middleware/test_gatekeeper.py | 2 +-
test/unit/common/middleware/test_slo.py | 217 +-
test/unit/common/middleware/test_tempauth.py | 45 +-
.../common/middleware/test_versioned_writes.py | 24 +
test/unit/common/missing_container_info.db | Bin 0 -> 17408 bytes
test/unit/common/ring/test_builder.py | 15 +-
test/unit/common/ring/test_composite_builder.py | 89 +-
test/unit/common/ring/test_ring.py | 22 +-
test/unit/common/ring/test_utils.py | 3 +
test/unit/common/test_constraints.py | 18 +-
test/unit/common/test_daemon.py | 8 +
test/unit/common/test_db.py | 393 +-
test/unit/common/test_db_replicator.py | 615 ++-
test/unit/common/test_direct_client.py | 70 +
test/unit/common/test_linkat.py | 6 +-
test/unit/common/test_memcached.py | 158 +-
test/unit/common/test_storage_policy.py | 9 +-
test/unit/common/test_swob.py | 16 +
test/unit/common/test_utils.py | 1586 ++++++-
test/unit/common/test_wsgi.py | 336 +-
test/unit/container/test_backend.py | 2532 ++++++++++-
test/unit/container/test_replicator.py | 1112 ++++-
test/unit/container/test_server.py | 1268 +++++-
test/unit/container/test_sharder.py | 4580 ++++++++++++++++++++
test/unit/container/test_updater.py | 26 +-
test/unit/helpers.py | 2 +-
test/unit/obj/test_auditor.py | 42 +-
test/unit/obj/test_diskfile.py | 116 +-
test/unit/obj/test_expirer.py | 582 ++-
test/unit/obj/test_reconstructor.py | 77 +-
test/unit/obj/test_replicator.py | 713 ++-
test/unit/obj/test_server.py | 303 +-
test/unit/obj/test_ssync_sender.py | 19 +-
test/unit/obj/test_updater.py | 466 +-
test/unit/proxy/controllers/test_base.py | 154 +-
test/unit/proxy/controllers/test_container.py | 1024 ++++-
test/unit/proxy/test_server.py | 289 +-
tools/test-setup.sh | 2 +-
tox.ini | 42 +-
295 files changed, 62272 insertions(+), 2672 deletions(-)
Requirements updates
--------------------
diff --git a/requirements.txt b/requirements.txt
index 1001be5..c115113 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5 +5 @@
-dnspython>=1.14.0 # http://www.dnspython.org/LICENSE
+dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE
@@ -9 +9,3 @@ netifaces>=0.5,!=0.10.0,!=0.10.1
-pastedeploy>=1.3.3
+PasteDeploy>=1.3.3
+lxml
+requests>=2.14.2 # Apache-2.0
diff --git a/test-requirements.txt b/test-requirements.txt
index 968571a..5f515bd 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -11,3 +10,0 @@ nosehtmloutput>=0.0.3 # Apache-2.0
-sphinx>=1.6.2 # BSD
-openstackdocstheme>=1.11.0 # Apache-2.0
-os-api-ref>=1.0.0 # Apache-2.0
@@ -18,0 +16,5 @@ reno>=1.8.0 # Apache-2.0
+python-openstackclient
+boto
+requests-mock>=1.2.0 # Apache-2.0
+fixtures>=3.0.0 # Apache-2.0/BSD
+keystonemiddleware>=4.17.0 # Apache-2.0
More information about the Release-announce
mailing list