[Openstack-security] [Bug 1613901] Re: String "..%c0%af" causes 500 errors in multiple locations in Keystone v3
Charles Neill
charles.neill at rackspace.com
Wed Sep 28 00:22:27 UTC 2016
This affects a number of other OpenStack projects in a similar way,
including:
- Neutron
- Cinder
- Glance
More projects may be affected that we are unaware of.
=====================================================
Example traceback from Neutron
=====================================================
2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted ('10.0.2.2', 50029) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)
File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76, in format
return logging.StreamHandler.format(self, record)
File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line 297, in format
return logging.Formatter.format(self, record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
msg = msg % self.args
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in as_text
bytes = self.as_bytes()
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in as_bytes
url = self.url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
url = self.path_url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
bpath_info = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid start byte
Logged from file catch_errors.py, line 41
2016-09-28 00:16:43.489 1218 INFO neutron.wsgi [req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016 00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984
=====================================================
Example traceback from Cinder
=====================================================
2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted ('10.0.2.2', 49862) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: <type 'exceptions.UnicodeDecodeError'> 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return req.get_response(self.application)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response = req.get_response(self._app)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response = self.app(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 817, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault "url": request.url})
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault url = self.path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault bpath_info = bytes_(self.path_info, self.url_encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return req.encget(key, encattr=encattr)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return val.decode(encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return codecs.utf_8_decode(input, errors, True)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault
2016-09-27 23:50:00.224 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
return app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/request_id.py", line 37, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 81, in __call__
return self._error(ex, req)
File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 56, in _error
msg_dict = dict(url=req.url, status=status)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
url = self.path_url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
bpath_info = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.228 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-
4f7d-b69c-36c385d0fa10 admin] 10.0.2.2 "POST /v2/..%c0%af/backups
HTTP/1.1" status: 500 len: 139 time: 0.0827849
=====================================================
Example traceback from Glance
=====================================================
2016-09-28 00:11:55.844 32495 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 123, in __call__
response = self.process_request(req)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 148, in __call__
return self.func(req, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/healthcheck/__init__.py", line 361, in process_request
if req.path != self._path:
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 485, in path
bpath = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte
2016-09-28 00:11:55.854 32495 INFO eventlet.wsgi.server [-] 10.0.2.2 - -
[28/Sep/2016 00:11:55] "GET /v2/images/..%c0%af/tags/..%c0%af HTTP/1.1"
500 139 0.043949
I'm not sure whether right approach here is to file a bug with webob (looks like there are several that haven't been resolved yet [1] [2]), or to fix this in each project's respective wsgi error-handling code.
[1] https://github.com/Pylons/webob/issues/115
[2] https://github.com/Pylons/webob/issues/161
** Bug watch added: github.com/Pylons/webob/issues #115
https://github.com/Pylons/webob/issues/115
** Bug watch added: github.com/Pylons/webob/issues #161
https://github.com/Pylons/webob/issues/161
** Summary changed:
- String "..%c0%af" causes 500 errors in multiple locations in Keystone v3
+ String "..%c0%af" causes 500 errors in multiple locations
** Also affects: neutron
Importance: Undecided
Status: New
** Also affects: glance
Importance: Undecided
Status: New
** Also affects: cinder
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1613901
Title:
String "..%c0%af" causes 500 errors in multiple locations
Status in Cinder:
New
Status in Glance:
New
Status in OpenStack Identity (keystone):
Confirmed
Status in neutron:
New
Status in OpenStack Security Advisory:
Won't Fix
Bug description:
While doing some testing on Keystone using Syntribos
(https://github.com/openstack/syntribos), our team (myself, Michael
Dong, Rahul U Nair, Vinay Potluri, Aastha Dixit, and Khanak Nangia)
noticed that we got 500 status codes when the string "..%c0%af" was
inserted in various places in the URL for different types of requests.
Here are some examples:
=========
DELETE /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
Content-Length: 0
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:04:27 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-238fd5a9-be45-41f2-893a-97b513b27af3
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
PATCH /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
Content-type: application/json
X-Auth-Token: [REDACTED]
Content-Length: 70
{"type": "--serialization-mime-type--", "blob": "--serialized-blob--"}
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:05:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-57a41600-02b4-4d2a-b3e9-40f7724d65f2
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
GET /v3/domains/0426ac1e48f642ef9544c2251e07e261/groups/..%c0%af/roles HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:07:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-02313f77-63c6-4aa8-a87e-e3d2a13ad6b7
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
I've marked this as a security issue as a precaution in case it turns
out that there is a more serious vulnerability underlying these
errors. We have no reason to suspect that there is a greater
vulnerability at this time, but given the many endpoints this seems to
affect, I figured caution was worthwhile since this may be a
framework-wide issue. Feel free to make this public if it is
determined not to be security-impacting.
Here is a (possibly incomplete) list of affected endpoints. Inserting
the string "..%c0%af" in any or all of the spots labeled "HERE" should
yield a 500 error. As you can see, virtually all v3 endpoints exhibit
this behavior.
=========
[GET|PATCH|DELETE] /v3/endpoints/[HERE]
[GET|PATCH] /v3/domains/[HERE]
GET /v3/domains/[HERE]/groups/[HERE]/roles
[HEAD|PUT|DELETE] /v3/domains/[HERE]/groups/[HERE]/roles/[HERE]
GET /v3/domains/[HERE]/users/[HERE]/roles
[HEAD|DELETE] /v3/domains/[HERE]/users/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/groups/[HERE]
[HEAD|PUT|DELETE] /v3/groups[HERE]/users/[HERE]
[POST|DELETE] /v3/keys/[HERE]
[GET|PATCH|DELETE] /v3/policies/[HERE]
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/endpoints/[HERE]
[GET|HEAD] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/policy
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/regions/[HERE]
[GET|PATCH|DELETE] /v3/projects/[HERE]
[DELETE|PATCH] /v3/projects/[HERE]/cascade
GET /v3/projects/[HERE]/groups/[HERE]/roles
GET /v3/projects/[HERE]/users/[HERE]/roles
[HEAD|PUT|DELETE] /v3/projects/[HERE]/groups/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/regions/[HERE]
[PATCH|DELETE] /v3/roles/[HERE]
[GET|PATCH|DELETE] /v3/services/[HERE]
[GET|PATCH|DELETE] /v3/users/[HERE]
GET /v3/users/[HERE]/groups
POST /v3/users/[HERE]/password
GET /v3/users/[HERE]/projects
GET /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/OS-OAUTH1/consumers/[HERE]
[GET|DELETE] /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1613901/+subscriptions
More information about the Openstack-security
mailing list