Open Stack

Having another look at that issue, it sounds like slow client shouldn't
be handled by OpenStack services but rather with a load balancer,
especially if the service is Internet facing.

I initially thought the problem would happen between the proxy and the
object server even after the client got disconnected, but this is not
the case since all the sockets are effectively released when the client
is disconnected.

Since there is already a Security Note in the process to cover load
balancers usage in front of OpenStack service, I suggest we close the
ossa task and triage this report as a class D according to the VMT
taxonomy ( https://security.openstack.org/vmt-process.html#incident-
report-taxonomy )

-- 
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1572719

Title:
  Client may hold socket open after ChunkWriteTimeout

Status in OpenStack Security Advisory:
  Incomplete
Status in OpenStack Object Storage (swift):
  Confirmed

Bug description:
  You can reproduce this by issuing a GET request for a few hundred MB
  file and never consuming the response, but keep the client socket
  open. Swift will log a 499 but the socket does not always close.

  ChunkWriteTimeout is meant to protect the proxy from a slow reading client:
    https://github.com/openstack/swift/blob/master/swift/proxy/controllers/base.py#L889-L905

  Sometimes when this exception is thrown there is still data in the process socket buffer, so when eventlet tries to close the socket it first flushes it:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L631
    https://hg.python.org/cpython/file/v2.7.11/Lib/SocketServer.py#l711

  The problem is that if the client is not consuming the socket buffer
  then that flush will wait forever; it's trying to write on a socket
  that just threw a timeout trying to write! The flush write is not
  protected by any timeout.

  As far as I can tell, this WRITE_TIMEOUT does nothing:
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L407

  wsgi.server() takes a socket_timeout that might be what we're after?
    https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L437-L440

  Even with socket_timeout, eventlet needs to be patched. This should be in a finally block:
    https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L636-L637

  All of this is probably mitigated by most operators setting an idle
  timeout in their load balancers, but I wanted to report it. Going
  directly to a proxy I was able to hold sockets open for long periods
  of time.

  I did the initial research on version 2.2.2 but I was able to
  reproduce on 2.7.0. I'm trying to translate links to master branch on
  github. I apologize in advance if it's not quite right.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1572719/+subscriptions