Open Stack

Opening this bug based on above comments

** Description changed:

- This issue is being treated as a potential security risk under embargo.
- Please do not make any public mention of embargoed (private) security
- vulnerabilities before their coordinated publication by the OpenStack
- Vulnerability Management Team in the form of an official OpenStack
- Security Advisory. This includes discussion of the bug or associated
- fixes in public forums such as mailing lists, code review systems and
- bug trackers. Please also avoid private disclosure to other individuals
- not already approved for access to this information, and provide this
- same reminder to those who are made aware of the issue prior to
- publication. All discussion should remain confined to this private bug
- report, and any proposed fixes should be added to the bug as
- attachments.
- 
  You can reproduce this by issuing a GET request for a few hundred MB
  file and never consuming the response, but keep the client socket open.
  Swift will log a 499 but the socket does not always close.
  
  ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
    https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905
  
  Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
    https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711
  
  The problem is that if the client is not consuming the socket buffer
  then that flush will wait forever; it's trying to write on a socket that
  just threw a timeout trying to write! The flush write is not protected
  by any timeout.
  
  As far as I can tell, this WRITE_TIMEOUT does nothing:
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407
  
  wsgi.server() takes a socket_timeout that might be what we're after?
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440
  
  Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637
  
  All of this is probably mitigated by most operators setting an idle
  timeout in their load balancers, but I wanted to report it. Going
  directly to a proxy I was able to hold sockets open for long periods of
  time.
  
  I did the initial research on version 2.2.2 but I was able to reproduce
  on 2.7.0. I'm trying to translate links to master branch on github. I
  apologize in advance if it's not quite right.

** Tags added: security

** Information type changed from Private Security to Public

-- 
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1572719

Title:
  Client may hold socket open after ChunkWriteTimeout

Status in OpenStack Security Advisory:
  Incomplete
Status in OpenStack Object Storage (swift):
  New

Bug description:
  You can reproduce this by issuing a GET request for a few hundred MB
  file and never consuming the response, but keep the client socket
  open. Swift will log a 499 but the socket does not always close.

  ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
    https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905

  Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
    https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711

  The problem is that if the client is not consuming the socket buffer
  then that flush will wait forever; it's trying to write on a socket
  that just threw a timeout trying to write! The flush write is not
  protected by any timeout.

  As far as I can tell, this WRITE_TIMEOUT does nothing:
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407

  wsgi.server() takes a socket_timeout that might be what we're after?
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440

  Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637

  All of this is probably mitigated by most operators setting an idle
  timeout in their load balancers, but I wanted to report it. Going
  directly to a proxy I was able to hold sockets open for long periods
  of time.

  I did the initial research on version 2.2.2 but I was able to
  reproduce on 2.7.0. I'm trying to translate links to master branch on
  github. I apologize in advance if it's not quite right.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1572719/+subscriptions