We exuberantly announce the release of:
swift 2.15.0: OpenStack Object Storage
This release is part of the pike release series.
Download the package from:
https://tarballs.openstack.org/swift/
For more details, please see below.
2.15.0 ^^^^^^
New Features
* Add Composite Ring Functionality
A composite ring comprises two or more component rings that are combined to form a single ring with a replica count equal to the sum of the component rings. The component rings are built independently, using distinct devices in distinct regions, which means that the dispersion of replicas between the components can be guaranteed.
Composite rings can be used for explicit replica placement and "replicated EC" for global erasure codes policies.
Composite rings support 'cooperative' rebalance which means that during rebalance all component rings will be consulted before a partition is moved in any component ring. This avoids the same partition being simultaneously moved in multiple components.
We do not yet have CLI tools for creating composite rings, but the functionality has been enabled in the ring modules to support this advanced functionality. CLI tools will be delivered in a subsequent release.
For further information see the docs
* The EC reconstructor process has been dramatically improved by adding support for multiple concurrent workers. Multiple processes are required to get high concurrency, and this change results in much faster rebalance times on servers with many drives.
Currently the default is still only one process, and no workers. Set "reconstructor_workers" in the "[object-reconstructor]" section to some whole number <= the number of devices on a node to get that many reconstructor workers.
* Add support to increase object ring partition power transparently to end users and with no cluster downtime. Increasing the ring part power allows for incremental adjustment to the upper bound of the cluster size. Please review the full docs for more information.
* Added support for per-policy proxy config options. This allows per- policy affinity options to be set for use with duplicated EC policies and composite rings. Certain options found in per-policy conf sections will override their equivalents that may be set in the [app:proxy-server] section. Currently the options handled that way are "sorting_method", "read_affinity", "write_affinity", "write_affinity_node_count", and "write_affinity_handoff_delete_count".
* Enabled versioned writes on Dynamic Large Objects (DLOs).
* Write-affinity aware object deletion
Previously, when deleting objects in multi-region swift deployment with write affinity configured, users always get 404 when deleting object before it's replicated to appropriate nodes.
Now Swift will use "write_affinity_handoff_delete_count" to define how many local handoff nodes should swift send request to get more candidates for the final response. The default value "auto" means Swift will calculate the number automatically based on the number of replicas and current cluster topology.
* Require that known-bad EC schemes be deprecated
Erasure-coded storage policies using "isa_l_rs_vand" and "nparity"
= 5 must be configured as deprecated, preventing any new containers
from being created with such a policy. This configuration is known to harm data durability. Any data in such policies should be migrated to a new policy. See See Launchpad bug 1639691 for more information.
* Optimize the Erasure Code reconstructor protocol to reduce IO load on servers.
* Fixed a bug where SSYNC would fail to replicate unexpired object.
* Fixed a bug in domain_remap when obj starts/ends with slash.
* Fixed a socket leak in copy middleware when a large object was copied.
* Fixed a few areas where the "swiftdir" option was not respected.
* swift-recon now respects storage policy aliases.
* cname_lookup middleware now accepts a "nameservers" config variable that, if defined, will be used for DNS lookups instead of the system default.
* Make mount_check option usable in containerized environments by adding a check for an ".ismount" file at the root directory of a device.
* Remove deprecated "vm_test_mode" option.
* The object and container server config option "slowdown" has been deprecated in favor of the new "objects_per_second" and "containers_per_second" options.
* The output of devices from "swift-ring-builder" has been reordered by region, zone, ip, and device.
* Imported docs content from openstack-manuals project.
* Various other minor bug fixes and improvements.
Changes in swift 2.14.0..2.15.0 -------------------------------
dfd0ea2 2.15.0 authors and changelog update 9155e6f added .htaccess redirect file for docs 56a18ac Add unit test for ObjectReconstructor.is_healthy 701a172 Add multiple worker processes strategy to reconstructor 93b8cf5 Small fixup for CooperativeRingBuilder 8c37305 Remove left over param d0bfd03 ready yet? nope, please wait! 31f4794 Close stdout *after* binding ports c211141 Add ever_rebalanced property to RingBuilder 71ec83f Ring rebalance respects co-builders' last_part_moves 5b10cf5 Add more structure to the deployment guide 35cf189 Stop using deprecated 'message' attribute in Exception 9c5628b Add reconstructor section to deployment guide c2e59b9 Make dict deletion idempotent in dump_recon_cache e1a2c28 Fix redundant UTF-8 check 95ff34d Add X-Copy-From-Account to api-ref 2410b61 Fix swiftdir option and usage of storage policy aliases 13a07aa Misc doc cleanup 63c4215 Fix rst for link 74700eb Fix StopIteration noise in obj/test_replicator.py 65b6b12 Add small note for handoffs_only acceleration 20a0d67 Clean up some assertions in test_reconstructor 239dfb5 moved install guide and removed tox env definition 44c63c6 Don't rehash primaries in reconstructor handoffs_only mode d112b7d First pass at admin-guide cleanup 4cb76a4 docs migration from openstack-manuals 006a378 Add license in swift code file bfb5759 Switch from oslosphinx to openstackdocstheme 58d7812 fix flakey time for test_default_sorted_output 14b07ca Untangle ServersPerPortStrategy.bind_ports 806b18c Extend device tier pretty print normalization 62509cc Improve debug logging from ring builder. 1eeb354 Fix the reST field raises in docstrings e11a38c Bind SAIO services on different loopback addresses 831eb6e Write-affinity aware object deletion 99a6d3b Test: Use assertIsNone() in unittest d9c4913 Make eventlet.tpool's thread count configurable in object server a1134e4 Order devices in the output of swift-ring-builder 3820e67 Add tests for Recon's object replication_time time unit 2cb74b1 Replaced assertTrue(False, msg) with fail(msg) 3cccd5a Make bin/swift-get-nodes executable a8bc94c Replace slowdown option with *_per_second option 2c3ac54 Require that known-bad EC schemes be deprecated e114066 Add support to increase object ring partition power 0b60b0b Close all versioned_writes subrequests' app_iters a7c8bec Fix a socket leak in copy middleware b5ee8c8 Ring doc cleanups 066f443 Follow up for affinity config per policy 4c7839d More Global EC doc updates 9665252 Update Global EC docs with reference to composite rings 188c07e Limit number of revert tombstone SSYNC requests 6687d2f Cleanup db replicator probetest c23b6db Use config_number method instead of node_id + 1 ee9458a Using assertIsNone() instead of assertEqual(None) 8d6eb36 Using assertIsNone() instead of assertEqual(None) 5d7b0d1 Delay port binding to reduce wait at process start d51ecb4 Remove threads_per_disk from object-server.conf manpages 420e73f Allow to configure the nameservers in cname_lookup 5ecf828 Follow-up for per-policy proxy configs 61bcd75 README: dedent the numbered list a3aba99 remove remote qualifier from release notes branch scanning f64fa46 Make probe tests work when policy-0 is EC c34ac98 Stop having Sphinx treat warnings as errors 37ba21f Add structure to storage policy configuration guide cf3c970 Trivial fix typos 227cef9 Add link from policies overview to per-policy proxy-server conf 675145e Remove deprecated vm_test_mode option d487bf7 Remove tempauth docs from deployment guide 45884c1 Enable per policy proxy config options 1b99180 Clean up how PatchPolicies works 5eeaa95 Make mount_check option usable in containerized environments f02ec4d Add read and write affinity options to deployment guide 4dc985a Small minor fixes for composite ring functionality 582af7c name_check: better test maximum_length d40031b Add Composite Ring Functionality cd2d359 Drop unnecessary dereferencing 843184f Sync metadata in 'rsync_then_merge' in db_replicator 3981e1e Remove links for EOLed releases 107ba79 Rewrite redirection in cname_lookup & domain_remap 37f6f25 Update multi-node install links 6db12b8 Fix domain_remap when obj starts/ends with slash 5dfc3a7 Open-code eventlet.listen() cd50160 Trivial fix typo and PEP 8 coding style violation e23f8d3 Trivial fix typo while reading doc 50357de Fix sporadic failure in TestObjectController.test_container_update_async 4c82395 Add message to account-creation assertion 511ac2e Use setUpModule instead of setup for module level unit test setup 172c4de Trivial fix warnings in docstring 88ca232 Trivial fix for decrypter docstrings 75290ec Fix sporadic failures in test_reconstructor.py 6c320b2 Stop including Connection header in EC GET responses 96bbd06 Remove the unused import code 7a9269b Fix unit test failing when swift.conf has default policy index >10 387ce13 Use LogRecord.msg instead of LogRecord.message in tests 93ca16c Fix install guide filter:authtoken config example 85d6cd3 Add Timestamp.now() helper a07f7dc Fix sporadic failure in TestAccountController unit test 2007257 Fix sporadic failure in test/unit/obj/test_server.py 38d3579 Fix SSYNC failing to replicate unexpired object 9e15eff Fix unit tests on i386 and other archs c740447 Move EC-specific unit test to EC Test class 381640c Fix (un)patch_policies a0fcca1 Do not sync suffixes when remote rejects reconstructor revert 3ad8773 Version DLOs, just like every other type of object 4fa536a Replace dict.iteritems with six.iteritems
Diffstat (except docs and test files) -------------------------------------
.mailmap | 2 + AUTHORS | 12 +- CHANGELOG | 111 ++ README.rst | 6 +- api-ref/source/parameters.yaml | 8 + api-ref/source/storage-object-services.inc | 1 + bin/swift-get-nodes | 6 + bin/swift-object-info | 5 + bin/swift-object-relinker | 39 + .../figures/objectstorage-accountscontainers.png | Bin 0 -> 33163 bytes .../admin/figures/objectstorage-buildingblocks.png | Bin 0 -> 49416 bytes .../admin/figures/objectstorage-partitions.png | Bin 0 -> 28638 bytes .../admin/figures/objectstorage-replication.png | Bin 0 -> 45780 bytes ...objectstorage-tenant-specific-image-storage.rst | 32 + .../install/controller-common_prerequisites.txt | 116 ++ .../finalize-installation-ubuntu-debian.rst | 80 ++ .../install/storage-install-ubuntu-debian.rst | 159 +++ etc/container-server.conf-sample | 6 +- etc/object-server.conf-sample | 53 +- etc/proxy-server.conf-sample | 74 +- etc/rsyncd.conf-sample | 2 +- etc/swift.conf-sample | 7 +- install-guide/source/conf.py | 301 ----- .../source/controller-common_prerequisites.txt | 116 -- install-guide/source/controller-include.txt | 84 -- install-guide/source/controller-install-debian.rst | 50 - install-guide/source/controller-install-obs.rst | 45 - install-guide/source/controller-install-rdo.rst | 50 - install-guide/source/controller-install-ubuntu.rst | 52 - install-guide/source/controller-install.rst | 18 - install-guide/source/edit_hosts_file.txt | 19 - install-guide/source/environment-networking.rst | 66 - install-guide/source/finalize-installation-obs.rst | 80 -- install-guide/source/finalize-installation-rdo.rst | 89 -- .../source/finalize-installation-ubuntu-debian.rst | 80 -- install-guide/source/finalize-installation.rst | 13 - install-guide/source/get_started.rst | 51 - install-guide/source/index.rst | 24 - install-guide/source/initial-rings.rst | 253 ---- install-guide/source/next-steps.rst | 10 - install-guide/source/storage-include1.txt | 41 - install-guide/source/storage-include2.txt | 41 - install-guide/source/storage-include3.txt | 42 - install-guide/source/storage-install-obs.rst | 139 -- install-guide/source/storage-install-rdo.rst | 155 --- .../source/storage-install-ubuntu-debian.rst | 159 --- install-guide/source/storage-install.rst | 16 - install-guide/source/verify.rst | 94 -- .../notes/2_15_0_release-0a05a011fb85a9c9.yaml | 102 ++ releasenotes/source/conf.py | 19 +- releasenotes/source/newton.rst | 2 +- releasenotes/source/ocata.rst | 2 +- setup.cfg | 2 +- swift/account/backend.py | 3 +- swift/account/reaper.py | 4 +- swift/account/server.py | 2 +- swift/account/utils.py | 3 +- swift/cli/info.py | 2 +- swift/cli/recon.py | 19 +- swift/cli/relinker.py | 159 +++ swift/cli/ring_builder_analyzer.py | 2 +- swift/cli/ringbuilder.py | 202 ++- swift/common/constraints.py | 7 +- swift/common/daemon.py | 210 ++- swift/common/db.py | 4 +- swift/common/db_replicator.py | 9 +- swift/common/direct_client.py | 3 +- swift/common/memcached.py | 12 +- swift/common/middleware/__init__.py | 36 + swift/common/middleware/bulk.py | 4 +- swift/common/middleware/cname_lookup.py | 56 +- swift/common/middleware/copy.py | 9 +- swift/common/middleware/crypto/crypto_utils.py | 2 +- swift/common/middleware/crypto/decrypter.py | 4 +- swift/common/middleware/domain_remap.py | 22 +- swift/common/middleware/list_endpoints.py | 2 +- swift/common/middleware/ratelimit.py | 10 +- swift/common/middleware/slo.py | 6 +- swift/common/middleware/tempauth.py | 276 ++-- swift/common/middleware/versioned_writes.py | 21 +- swift/common/middleware/x_profile/html_viewer.py | 2 +- swift/common/request_helpers.py | 10 +- swift/common/ring/builder.py | 157 ++- swift/common/ring/composite_builder.py | 735 +++++++++++ swift/common/ring/ring.py | 37 +- swift/common/ring/utils.py | 19 +- swift/common/storage_policy.py | 26 +- swift/common/swob.py | 7 +- swift/common/utils.py | 116 +- swift/common/wsgi.py | 23 +- swift/container/backend.py | 9 +- swift/container/reconciler.py | 2 +- swift/container/replicator.py | 5 +- swift/container/server.py | 2 +- swift/container/updater.py | 21 +- swift/obj/diskfile.py | 133 +- swift/obj/reconstructor.py | 270 +++- swift/obj/replicator.py | 21 +- swift/obj/server.py | 48 +- swift/obj/ssync_receiver.py | 4 +- swift/obj/updater.py | 21 +- swift/proxy/controllers/base.py | 39 +- swift/proxy/controllers/container.py | 3 +- swift/proxy/controllers/obj.py | 156 ++- swift/proxy/server.py | 229 +++- test-requirements.txt | 5 +- test/__init__.py | 15 + test/functional/__init__.py | 43 +- test/functional/test_versioned_writes.py | 69 +- test/probe/__init__.py | 18 + test/probe/brain.py | 2 +- test/probe/common.py | 9 +- test/probe/test_container_merge_policy_index.py | 9 +- test/probe/test_db_replicator.py | 152 +++ test/probe/test_object_metadata_replication.py | 22 +- test/probe/test_object_partpower_increase.py | 199 +++ test/probe/test_reconstructor_revert.py | 75 +- test/probe/test_replication_servers_working.py | 2 +- test/unit/__init__.py | 73 +- test/unit/account/test_backend.py | 86 +- test/unit/account/test_reaper.py | 13 +- test/unit/account/test_server.py | 2 +- test/unit/cli/test_default_output.stub | 2 +- test/unit/cli/test_default_output_id_assigned.stub | 11 + test/unit/cli/test_default_sorted_output.stub | 10 + test/unit/cli/test_ipv6_output.stub | 2 +- test/unit/cli/test_recon.py | 31 +- test/unit/cli/test_relinker.py | 172 +++ test/unit/cli/test_ringbuilder.py | 87 +- test/unit/common/middleware/test_bulk.py | 4 +- test/unit/common/middleware/test_cname_lookup.py | 163 ++- test/unit/common/middleware/test_copy.py | 10 +- test/unit/common/middleware/test_dlo.py | 12 +- test/unit/common/middleware/test_domain_remap.py | 100 +- test/unit/common/middleware/test_except.py | 12 +- test/unit/common/middleware/test_formpost.py | 66 +- test/unit/common/middleware/test_keystoneauth.py | 2 +- test/unit/common/middleware/test_list_endpoints.py | 2 +- test/unit/common/middleware/test_memcache.py | 2 +- test/unit/common/middleware/test_name_check.py | 13 +- test/unit/common/middleware/test_proxy_logging.py | 4 +- test/unit/common/middleware/test_ratelimit.py | 13 +- test/unit/common/middleware/test_slo.py | 8 +- test/unit/common/middleware/test_tempauth.py | 31 +- test/unit/common/middleware/test_tempurl.py | 10 +- .../common/middleware/test_versioned_writes.py | 70 +- test/unit/common/ring/test_builder.py | 251 +++- test/unit/common/ring/test_composite_builder.py | 1173 +++++++++++++++++ test/unit/common/ring/test_utils.py | 17 +- test/unit/common/test_bufferedhttp.py | 6 +- test/unit/common/test_constraints.py | 42 +- test/unit/common/test_container_sync_realms.py | 10 +- test/unit/common/test_daemon.py | 207 ++- test/unit/common/test_db.py | 4 +- test/unit/common/test_db_replicator.py | 95 +- test/unit/common/test_direct_client.py | 10 +- test/unit/common/test_header_key_dict.py | 2 +- test/unit/common/test_manager.py | 10 +- test/unit/common/test_memcached.py | 8 +- test/unit/common/test_request_helpers.py | 2 +- test/unit/common/test_storage_policy.py | 27 +- test/unit/common/test_swob.py | 40 +- test/unit/common/test_utils.py | 145 +- test/unit/common/test_wsgi.py | 58 +- test/unit/container/test_backend.py | 205 +-- test/unit/container/test_reconciler.py | 18 +- test/unit/container/test_server.py | 15 +- test/unit/container/test_sync.py | 16 +- test/unit/container/test_updater.py | 50 +- test/unit/helpers.py | 25 +- test/unit/obj/common.py | 7 +- test/unit/obj/test_auditor.py | 121 +- test/unit/obj/test_diskfile.py | 169 ++- test/unit/obj/test_reconstructor.py | 1327 +++++++++++++++++-- test/unit/obj/test_replicator.py | 45 +- test/unit/obj/test_server.py | 82 +- test/unit/obj/test_ssync.py | 101 +- test/unit/obj/test_ssync_receiver.py | 47 +- test/unit/obj/test_updater.py | 50 +- test/unit/proxy/controllers/test_container.py | 2 +- test/unit/proxy/controllers/test_obj.py | 220 +++- test/unit/proxy/test_mem_server.py | 13 +- test/unit/proxy/test_server.py | 1382 +++++++++++++++----- tox.ini | 3 - 261 files changed, 14273 insertions(+), 4587 deletions(-)
Requirements updates --------------------
diff --git a/test-requirements.txt b/test-requirements.txt index 5b944ec..968571a 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -11,3 +11,2 @@ nosehtmloutput>=0.0.3 # Apache-2.0 -openstackdocstheme>=1.5.0 # Apache-2.0 -oslosphinx>=4.7.0 # Apache-2.0 -sphinx>=1.5.1 # BSD +sphinx>=1.6.2 # BSD +openstackdocstheme>=1.11.0 # Apache-2.0