[release-announce] swift 2.26.0 (victoria)

no-reply at openstack.org no-reply at openstack.org
Fri Sep 18 19:36:03 UTC 2020


We joyfully announce the release of:

swift 2.26.0: OpenStack Object Storage

This release is part of the victoria release series.

The source is available from:

    https://opendev.org/openstack/swift

Download the package from:

    https://tarballs.openstack.org/swift/

Please report issues through:

    https://bugs.launchpad.net/swift/+bugs

For more details, please see below.

2.26.0
^^^^^^


New Features
************

* Extend concurrent reads to erasure coded policies. Previously, the
  options "concurrent_gets" and "concurrency_timeout" only applied to
  replicated policies.

* Add a new "concurrent_ec_extra_requests" option to allow the proxy
  to make some extra backend requests immediately. The proxy will
  respond as soon as there are enough responses available to
  reconstruct.

* The concurrent read options ("concurrent_gets",
  "concurrency_timeout", and "concurrent_ec_extra_requests") may now
  be configured per storage-policy.

* Replication servers can now handle all request methods. This
  allows ssync to work with a separate replication network.

* All background daemons now use the replication network. This
  allows better isolation between external, client-facing traffic and
  internal, background traffic. Note that during a rolling upgrade,
  replication servers may respond with "405 Method Not Allowed". To
  avoid this, operators should remove the config option
  "replication_server = true" from their replication servers; this
  will allow them to handle all request methods before upgrading.

* S3 API improvements:

  * Fixed some SignatureDoesNotMatch errors when using the AWS .NET
    SDK.

  * Add basic read support for object tagging. This improves
    compatibility with AWS CLI version 2. Write support is not yet
    implemented, so the tag set will always be empty.

  * CompleteMultipartUpload requests may now be safely retried.

  * Improved quota-exceeded error messages.

  * Improved logging and statsd metrics. Be aware that this will
    cause an increase in the proxy-logging statsd metrics emited for
    S3 responses. However, this should more accurately reflect the
    state of the system.

  * S3 requests are now less demanding on the container layer.

* Servers now open one listen socket per worker, ensuring each
  worker serves roughly the same number of concurrent connections.

* Server workers may now be gracefully terminated via "SIGHUP" or
  "SIGUSR1". The parent process will then spawn a fresh worker.

* Allow proxy-logging middlewares to be configured more
  independently.

* Improve performance when increasing partition power.


Known Issues
************

* In a rolling upgrade from liberasurecode 1.5.0 or earlier to 1.6.0
  or later, object-servers may quarantine newly-written data, leading
  to availability issues or even data loss. See bug 1886088 for more
  information, including how to determine whether you are affected.
  Several mitigations are available to operators:

  * If proxy and object layers can be upgraded independently and
    proxies can be upgraded quickly:

    1. Stop and disable the object-reconstructor before upgrading.
       This ensures no upgraded object server starts writing new
       fragments that old object servers would quarantine.

    2. Upgrade liberasurecode on all object servers. Object servers
       can now read both old and new fragments.

    3. Upgrade liberasurecode on all proxy servers. Newly-written
       data will now use new fragments. Note that not-yet-upgraded
       proxies will not be able to read these newly-written fragments
       but will instead respond "500 Internal Server Error".

    4. After upgrading, re-enable and restart the object-
       reconstructor.

  * If your users can tolerate it, consider a read-only rolling
    upgrade. Before upgrading, enable the read-only middleware
    cluster-wide to prevent new writes during the upgrade.
    Additionally, stop and disable the object-reconstructor as above.
    Upgrade normally, then disable the read-only middleware and re-
    enable and restart the object-reconstructor.

  * Avoid upgrading liberasurecode until swift and liberasurecode
    better-support a rolling upgrade. Swift remains compatible with
    liberasurecode 1.5.0 and earlier.

  Note: Ubuntu 18.04 and RDO's CentOS 7 repos package liberasurecode
    1.5.0, while Ubuntu 20.04 and RDO's CentOS 8 repos currently
    package liberasurecode 1.6.0 or 1.6.1. Take care when upgrading
    major distro versions!


Upgrade Notes
*************

* **If your cluster has encryption enabled and is still running
  Swift under Python 2**, we recommend upgrading Swift *before*
  transitioning to Python 3. Otherwise, new writes to objects with
  non-ASCII characters in their paths may result in corrupted
  downloads when read from a proxy-server still running old swift on
  Python 2. See bug 1888037 for more information. Note that new tags
  including a fix for the bug are planned for all maintained stable
  branches; upgrading to any one of those should be sufficient to
  ensure a smooth upgrade to the latest Swift.

* The above bug was caused by a difference in string types that
  resulted in ambiguity when decrypting. To prevent the ambiguity for
  new data, set "meta_version_to_write = 3" in your keymaster
  configuration *after* upgrading all proxy servers.

  If upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set
  "meta_version_to_write = 1" in your keymaster configuration *prior*
  to upgrading.

  See the provided "keymaster.conf-sample" for more information about
  this setting.

* **If your cluster is configured with a separate replication
  network**, note that background daemons will switch to using this
  network for all traffic. If your account, container, or object
  replication servers are configured with "replication_server = true",
  these daemons may log a flood of "405 Method Not Allowed" messages
  during a rolling upgrade. To avoid this, comment out the option and
  restart replication servers before upgrading.


Bug Fixes
*********

* Python 3 bug fixes:

  * Fixed an error when reading encrypted data that was written
    while running Python 2 for a path that includes non-ASCII
    characters.

  * Object expiration respects the
    "expiring_objects_container_divisor" config option.

  * "fallocate_reserve" may be specified as a percentage in more
    places.

  * The ETag-quoting middleware no longer raises TypeErrors.

* Sharding improvements:

  * Prevent object updates from auto-creating shard containers. This
    ensures more consistent listings for sharded containers during
    rebalances.

  * Deleted shard containers are no longer considered root
    containers. This prevents unnecessary sharding audit failures and
    allows the deleted shard database to actually be unlinked.

  * "swift-container-info" now summarizes shard range information.
    Pass "-v"/"--verbose" if you want to see all of them.

  * Improved container-sharder stat reporting to reduce load on root
    container databases.

  * Don't inject shard ranges when user quits.

* During rebalances, clients should no longer get 404s for data that
  exists but whose replicas are overloaded.

* Improved cache management for account and container responses.

* Allow operators to pass either raw or URL-quoted paths to "swift-
  get-nodes". Notably, this allows "swift-get-nodes" to work with the
  reserved namespace used for object versioning.

* Container read ACLs now work with object versioning. This only
  allows access to the most-recent version via an unversioned URL.

* Improved how containers reclaim deleted rows to reduce locking and
  object update throughput.

* Large object reads log fewer client disconnects.

* Allow ratelimit to be placed multiple times in a proxy pipeline,
  such as both before s3api and auth (to handle swift requests without
  needing to make an auth decision) and after (to limit S3 requests).

* Shuffle object-updater work. This somewhat reduces the impact a
  single overloaded database has on other containers' listings.

* Fix a proxy-server error when retrieving erasure coded data when
  there are durable fragments but not enough to reconstruct.

* Fix an error in the proxy server when finalizing data.

* Various other minor bug fixes and improvements.

Changes in swift 2.25.0..2.26.0
-------------------------------

d6c6ab764 Authors/ChangeLog for 2.26.0
a8d214626 xlo: Drain error responses
f8a9a6e11 s3api: Make quota-exceeded errors more obvious
ae8a89f04 gate: Make rolling upgrade job work with either 60xx or 62xx ports
d5625abf6 proxy: Include thread_locals when spawning _fragment_GET_request
754defc39 Client should retry when there's just one 404 and a bunch of errors
6e9e41d9b Remove some useless swob.Request attr setting
b7b45eadc ec: Close down some unused responses more quickly
f043aedec Make all concurrent_get options per-policy
8ea227a0e Suppress CryptographyDeprecationWarnings
ee693265f swift-init: Don't expose misleading commands
eefe35755 docs: Clean up some formatting around using servers_per_port
8f60e0a26 Extend concurrent_gets to EC GET requests
3f5e712be wsgi: Allow workers to gracefully exit
452db14a0 Bind a new socket per-worker
829a0d9a7 [goal] Migrate testing to ubuntu focal
08eacf68d Fix a typo in the explanatory note
5b2c846c6 Start to decouple Object GET path
2a6dfae2f Allow direct and internal clients to use the replication network
7d429318d py3: Work with proper native string paths in crypto meta
3a6e85d9e s3api: Allow lower-cased region name for AWS .NET SDK compatibility
e10af5d26 docs: Clarify request_time in storage logs
770cc287a Treat add/remove/rebalance during part-power increase as error
cb67f9472 container-sharding: Stable overlap order
9eb81f6e6 Allow replication servers to handle all request methods
0dbf3d0a9 Actually interpolate error message
67e3830ab swift-container-info: Show shard ranges summary
fa5bad279 Decouple proxy_server tests
1c9154764 Use latest pip in gate jobs
314347a3c Update SAIO & docker image to use 62xx ports
c31e30ec2 docs: Improve replication-network remakerings
5bd95cf2b probe tests: Get rid of `server` arg for device_dir() and storage_dir()
907942eb4 Stop syncing empty suffixes list
a5ec38326 Add a reminder to configure firewall at storage nodes
36bd21488 Address a sharder/replicator race
a4cf508a3 Identify deleted shards as shards
b2efd185c s3api: Use swift.backend_path to proxy-log s3api requests
45be775eb Get rid of memcache attr on proxy-server app
7753eff66 py3: Stop munging RAW_PATH_INFO
2e001431f s3api: Don't do naive HEAD request for auth
5c087ad7b Fix up curl commands when object name has unix dir characters
02548717a s3api: Allow CompleteMultipartUpload requests to be retried
e91a3d73a Remove lxml deprecated methods
5eb677fe5 Bump up probe test timeout
51a587ed8 Use ensure-pip role
f5c25d7c6 Clean up some proxy tests
7be7cc966 proxy: Stop killing memcache entries on 5xx responses
9bc8c8258 py3: Fix expirer container generation
6afefe1ad s3api: Add basic support for ?tagging requests
2ffe598f4 proxy-logging: Be able to configure log_route
79811df34 Use ini_file to update timeout instead of crudini
2854eddb4 py3: (Better) fix percentages in configs
e4586fdcd memcached: Plumb logger into MemcacheRing
1dfa41dad swift-get-nodes: Allow users to specify either quoted or unquoted paths
1b6c8f7fd Remove etag-quoter from 2.25.0 release notes
11dd0da29 tests: Bump up timeout for unit and in-process func tests
fc731198a Quiet eventlet exceptions in test
ce4c0fb14 Don't auto-create shard containers
3d105b623 Switch to newer openstackdocstheme and reno versions
019bade19 Remove <py3.5 dependencies from requirements.txt
738514c16 docs: s3api doesn't support tagging
20c6bdb71 Enable s3api and staticweb tests across all func tests
99947150d func tests: work with etag-quoter on by default
ede9dad9f Better functest quarantine cleanup
cedec8c5e Latch shard-stat reporting
60f052f69 dsvm: Run service-user tests under keystoneauth
984b57a87 tests: Skip s3api copy-version test when OV not enabled
3aa4e3ec8 Remove swift-dsvm-functional-py3 job
650272eac Imported Translations from Zanata
fa768b434 Simplify wsgify()
a2feefb04 dsvm: Use devstack's s3api "service"
5034916c1 Skip tests when only changing README.rst, CONTRIBUTING.rst, etc.
78cce72f8 Ussuri contrib docs community goal
73f0b143d dsvm: Run service-user tests under tempauth
63b16a368 dsvm: Run domain_remap tests with tempauth
a8e03f42e versioning: Have versioning symlinks make pre-auth requests to reserved container
63e02fa9f Test for Versioned Object If-Match
bb9b0326f swift-dsvm: Enable s3api
aab45880f Breakup reclaim into batches
f25705ee1 Add py38 package metadata
67598b3a4 Bump up timeouts for DSVM and probe test jobs
1db11df4f ratelimit: Allow multiple placements
f4bc95150 fix s3api functional tests
1358d3732 versioning: Use CONTAINER_LISTING_LIMIT when doing container listings
971023e4c replication: Allow databases_per_second to be a float
0fd23ee54 Fix pep8 job
dee98a74d updater: Shuffle suffixes so we don't keep hitting the same failures
f57d4cfa7 object-updater: Ignore ENOENT when trying to unlink stale pending files
fe74ec048 py27: Suppress UnicodeWarnings in ShardRange setters
630c9ef80 probe tests: Work when fronted by a TLS terminator
0f2da2aee Use local py2 upper-constraints
695295007 Switch to new grenade job name
cb37d3acd docs: Clean up mount options
4c8512afb Use separate name for HeaderKeyDict var vs list of response headers
1af995f0e s3api: Check whether versioning is enabled more
f3c19459c Imported Translations from Zanata
69b8165cd obj: _finalize_durable may succeed even when data file is missing
a9c6ad7e2 Add stable/ussuri upgrade job
9581254e6 swift-dsvm: Enable more middlewares
dbeaabe54 Add experimental job to upgrade from master
e84d531b9 py3: Make etag-quoter work
63e6c91ab Make rolling-upgrade job non-voting
72a617752 func tests: Allow test_etag_quoter to be run multiple times
d0f0d1d4f sharding: Add probe test that exercises swift-manage-shard-ranges
c7fe823c0 Update master for stable/ussuri
eae27412d sharding: Don't inject shard ranges when user says quit
3061ec803 relinker: Improve performance by limiting I/O


Diffstat (except docs and test files)
-------------------------------------

.zuul.yaml                                         | 172 ++--
AUTHORS                                            |   4 +
CHANGELOG                                          | 135 +++-
CONTRIBUTING.rst                                   |  10 +
REVIEW_GUIDELINES.rst                              |   2 +-
api-ref/source/conf.py                             |  44 +-
bin/swift-container-info                           |   4 +
bin/swift-get-nodes                                |   2 +
bin/swift-object-relinker                          |   2 +
.../install/storage-install-ubuntu-debian.rst      |   4 +-
etc/account-server.conf-sample                     |  14 +-
etc/container-server.conf-sample                   |  14 +-
etc/keymaster.conf-sample                          |  27 +
etc/object-server.conf-sample                      |  14 +-
etc/proxy-server.conf-sample                       |  52 +-
etc/rsyncd.conf-sample                             |  16 +-
examples/apache2/account-server.template           |   2 +-
examples/apache2/container-server.template         |   2 +-
examples/apache2/object-server.template            |   2 +-
lower-constraints.txt                              |   5 -
py2-constraints.txt                                |  72 ++
.../notes/2_26_0_release-6548eadcba544f72.yaml     | 216 +++++
releasenotes/source/conf.py                        |  14 +-
releasenotes/source/index.rst                      |   2 +
.../locale/en_GB/LC_MESSAGES/releasenotes.po       | 882 ++++++++++++++++++++-
releasenotes/source/ussuri.rst                     |   6 +
requirements.txt                                   |  11 +-
roles/additional-keystone-users/tasks/main.yaml    | 134 ++++
roles/additional-tempauth-users/tasks/main.yaml    |  47 ++
roles/dsvm-additional-middlewares/tasks/main.yaml  |  70 ++
setup.cfg                                          |   1 +
swift/__init__.py                                  |   7 +
swift/account/reaper.py                            |  10 +-
swift/cli/info.py                                  |  65 +-
swift/cli/manage_shard_ranges.py                   |   4 +-
swift/cli/recon.py                                 |  34 +-
swift/cli/relinker.py                              | 181 ++++-
swift/cli/ringbuilder.py                           |   8 +-
swift/common/base_storage_server.py                |  29 +-
swift/common/db.py                                 |  53 +-
swift/common/db_replicator.py                      |   2 +-
swift/common/direct_client.py                      |  33 +-
swift/common/internal_client.py                    |   7 +-
swift/common/manager.py                            |  98 ++-
swift/common/memcached.py                          |  21 +-
swift/common/middleware/crypto/keymaster.py        |  61 +-
swift/common/middleware/dlo.py                     |   7 +-
swift/common/middleware/etag_quoter.py             |  10 +-
swift/common/middleware/list_endpoints.py          |   6 +-
swift/common/middleware/memcache.py                |   5 +-
swift/common/middleware/proxy_logging.py           |  15 +-
swift/common/middleware/ratelimit.py               |   4 +
.../middleware/s3api/controllers/__init__.py       |   3 +
.../middleware/s3api/controllers/multi_upload.py   |  91 ++-
swift/common/middleware/s3api/controllers/obj.py   |  24 +-
.../common/middleware/s3api/controllers/tagging.py |  57 ++
swift/common/middleware/s3api/etree.py             |   2 +-
swift/common/middleware/s3api/s3api.py             |   2 +
swift/common/middleware/s3api/s3request.py         |  69 +-
swift/common/middleware/s3api/utils.py             |   4 -
swift/common/middleware/slo.py                     |  14 +-
swift/common/middleware/symlink.py                 |  31 +-
swift/common/middleware/tempauth.py                |   2 -
.../versioned_writes/object_versioning.py          |   6 +-
swift/common/request_helpers.py                    |  30 +-
swift/common/ring/builder.py                       |  16 +-
swift/common/swob.py                               |  27 +-
swift/common/utils.py                              | 127 ++-
swift/common/wsgi.py                               | 394 ++++-----
swift/container/backend.py                         | 118 ++-
swift/container/reconciler.py                      |  19 +-
swift/container/server.py                          |  44 +-
swift/container/sharder.py                         |  43 +-
swift/container/sync.py                            |   7 +-
swift/locale/de/LC_MESSAGES/swift.po               |   9 +-
swift/obj/diskfile.py                              |  21 +-
swift/obj/expirer.py                               |   3 +-
swift/obj/replicator.py                            |   3 +
swift/obj/updater.py                               |  13 +-
swift/proxy/controllers/account.py                 |   2 +-
swift/proxy/controllers/base.py                    | 129 +--
swift/proxy/controllers/container.py               |   2 +-
swift/proxy/controllers/obj.py                     | 823 ++++++++++++++++---
swift/proxy/server.py                              |  58 +-
test/__init__.py                                   |   6 +
test/functional/__init__.py                        |  29 +-
test/functional/s3api/__init__.py                  |  19 +-
test/functional/s3api/s3_test_client.py            |  40 +-
test/functional/s3api/test_acl.py                  |   6 +-
test/functional/s3api/test_bucket.py               |  16 +-
test/functional/s3api/test_multi_delete.py         |   2 +-
test/functional/s3api/test_multi_upload.py         |  66 +-
test/functional/s3api/test_object.py               |  10 +-
test/functional/s3api/test_service.py              |   2 +-
test/functional/s3api/test_versioning.py           |   4 +-
test/functional/swift_test_client.py               |   3 +
test/functional/test_object.py                     |  21 +-
test/functional/test_object_versioning.py          | 110 ++-
test/functional/test_slo.py                        |  31 +-
test/functional/test_symlink.py                    |  79 +-
test/functional/test_versioned_writes.py           |  11 +-
test/probe/__init__.py                             |   8 +
test/probe/brain.py                                |   3 +-
test/probe/common.py                               |  18 +-
.../probe/test_account_get_fake_responses_match.py |  12 +-
test/probe/test_container_failures.py              |  31 +-
test/probe/test_object_async_update.py             |   7 +-
test/probe/test_object_failures.py                 |   4 +-
test/probe/test_object_handoff.py                  |  19 +-
test/probe/test_object_metadata_replication.py     |   3 +-
test/probe/test_object_partpower_increase.py       |   4 +-
test/probe/test_reconstructor_rebuild.py           |   6 +-
test/probe/test_reconstructor_revert.py            |  20 +-
test/probe/test_sharder.py                         |  96 ++-
test/probe/test_signals.py                         |  72 +-
test/sample.conf                                   |   5 +
test/unit/__init__.py                              |  51 +-
test/unit/account/test_backend.py                  |  66 ++
test/unit/account/test_reaper.py                   |   6 +-
test/unit/account/test_server.py                   |  11 +-
test/unit/cli/test_info.py                         | 241 ++++--
test/unit/cli/test_manage_shard_ranges.py          |  21 +-
test/unit/cli/test_relinker.py                     | 218 ++++-
test/unit/cli/test_ringbuilder.py                  |   6 +-
.../common/middleware/crypto/test_keymaster.py     | 227 +++++-
test/unit/common/middleware/s3api/__init__.py      |  12 +
.../common/middleware/s3api/test_multi_upload.py   | 185 ++++-
test/unit/common/middleware/s3api/test_obj.py      | 135 +++-
test/unit/common/middleware/s3api/test_s3_acl.py   |  16 +-
test/unit/common/middleware/s3api/test_s3api.py    |  29 +-
.../unit/common/middleware/s3api/test_s3request.py |   4 +-
.../common/middleware/s3api/test_versioning.py     |   2 +-
test/unit/common/middleware/test_copy.py           |   4 +-
test/unit/common/middleware/test_dlo.py            |  29 +-
test/unit/common/middleware/test_proxy_logging.py  |  35 +-
test/unit/common/middleware/test_ratelimit.py      | 114 +--
test/unit/common/middleware/test_slo.py            |  17 +-
test/unit/common/middleware/test_symlink.py        |  50 ++
test/unit/common/ring/test_builder.py              |   2 +-
test/unit/common/test_base_storage_server.py       |  25 +-
test/unit/common/test_db.py                        |   4 +-
test/unit/common/test_db_replicator.py             |   6 +-
test/unit/common/test_direct_client.py             |  55 ++
test/unit/common/test_internal_client.py           |  73 +-
test/unit/common/test_memcached.py                 |  79 +-
test/unit/common/test_request_helpers.py           |  13 +
test/unit/common/test_swob.py                      |  16 +
test/unit/common/test_utils.py                     | 177 ++++-
test/unit/common/test_wsgi.py                      | 311 ++++----
test/unit/container/test_backend.py                | 314 +++++++-
test/unit/container/test_reconciler.py             |   1 +
test/unit/container/test_server.py                 | 119 ++-
test/unit/container/test_sharder.py                |  51 +-
test/unit/obj/test_diskfile.py                     |  86 ++
test/unit/obj/test_replicator.py                   |  37 +-
test/unit/obj/test_server.py                       |  24 +-
test/unit/proxy/controllers/test_account.py        |  48 +-
test/unit/proxy/controllers/test_base.py           | 206 ++++-
test/unit/proxy/controllers/test_container.py      |  53 +-
test/unit/proxy/controllers/test_obj.py            | 684 +++++++++++++++-
test/unit/proxy/test_server.py                     | 382 +++++----
test/unit/proxy/test_sysmeta.py                    |  82 +-
tools/playbooks/common/install_dependencies.yaml   |   8 +-
tools/playbooks/dsvm/pre.yaml                      |   3 +
.../multinode_setup/configure_loopback.yaml        |   2 +-
tools/playbooks/multinode_setup/make_rings.yaml    |  10 +
.../templates/make_multinode_rings.j2              |  40 +-
.../saio_single_node_setup/setup_saio.yaml         |  12 +-
tox.ini                                            |  37 +-
204 files changed, 8389 insertions(+), 2342 deletions(-)


Requirements updates
--------------------

diff --git a/requirements.txt b/requirements.txt
index d4c994af2..223fc617b 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5 +4,0 @@
-dnspython>=1.15.0;python_version=='2.7' # http://www.dnspython.org/LICENSE
@@ -16 +15,9 @@ cryptography>=2.0.2                     # BSD/Apache-2.0
-ipaddress>=1.0.16;python_version<'3.3'  # PSF
+
+# For python 2.7, the following requirements are needed; they are not
+# included since the requirments-check check will fail otherwise since
+# global requirements do not support these anymore.
+# Fortunately, these packages come in as dependencies from others and
+# thus the py27 jobs still work.
+#
+# dnspython>=1.15.0;python_version=='2.7' # http://www.dnspython.org/LICENSE
+# ipaddress>=1.0.16;python_version<'3.3'  # PSF






More information about the Release-announce mailing list