Open Stack

Reviewed:  https://review.openstack.org/224941
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Submitter: Jenkins
Branch:    stable/kilo

commit 5f86d05c50e02a895093d4cb6e4285dce4ba93a2
Author: yangxurong <yangxurong at huawei.com>
Date:   Mon Mar 30 14:39:17 2015 +1000

    Eventlet green threads not released back to pool
    
    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.
    
    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.
    
    Add a parameter to take advantage of the new(ish) eventlet socket timeout
    behaviour. Allows closing idle client connections after a period of
    time, eg:
    
    $ time nc localhost 8776
    real    1m0.063s
    Setting 'client_socket_timeout = 0' means do not timeout.
    
    DocImpact:
    Added wsgi_keep_alive option (default=True).
    Added client_socket_timeout option (default=900).
    SecurityImpact
    
    Change-Id: I303d87addeed8b103eeb26dbcc48e3acce06ee6a
    Closes-Bug: #1361360
    (cherry picked from commit 2be72f42f9debf9c2f186212dd27cb4f860d3399)


** Changed in: heat/kilo
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1361360

Title:
  Eventlet green threads not released back to the pool leading to
  choking of new requests

Status in Cinder:
  Fix Released
Status in Cinder icehouse series:
  Fix Released
Status in Cinder juno series:
  Fix Released
Status in Glance:
  Fix Released
Status in Glance icehouse series:
  Fix Committed
Status in heat:
  Fix Released
Status in heat kilo series:
  Fix Committed
Status in Keystone:
  Fix Released
Status in Keystone icehouse series:
  Confirmed
Status in Keystone juno series:
  Fix Committed
Status in Keystone kilo series:
  Fix Released
Status in Manila:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron icehouse series:
  Fix Released
Status in neutron juno series:
  Fix Committed
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  Fix Released
Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Security Notes:
  Won't Fix
Status in Sahara:
  Confirmed

Bug description:
  Currently reproduced  on Juno milestone 2. but this issue should be
  reproducible in all releases since its inception.

  It is possible to choke OpenStack API controller services using
  wsgi+eventlet library by simply not closing the client socket
  connection. Whenever a request is received by any OpenStack API
  service for example nova api service, eventlet library creates a green
  thread from the pool and starts processing the request. Even after the
  response is sent to the caller, the green thread is not returned back
  to the pool until the client socket connection is closed. This way,
  any malicious user can send many API requests to the API controller
  node and determine the wsgi pool size configured for the given service
  and then send those many requests to the service and after receiving
  the response, wait there infinitely doing nothing leading to
  disrupting services for other tenants. Even when service providers
  have enabled rate limiting feature, it is possible to choke the API
  services with a group (many tenants) attack.

  Following program illustrates choking of nova-api services (but this
  problem is omnipresent in all other OpenStack API Services using
  wsgi+eventlet)

  Note: I have explicitly set the wsi_default_pool_size default value to 10 in order to reproduce this problem in nova/wsgi.py.
  After you run the below program, you should try to invoke API
  ============================================================================================
  import time
  import requests
  from multiprocessing import Process

  def request(number):
     #Port is important here
     path = 'http://127.0.0.1:8774/servers'
      try:
          response = requests.get(path)
          print "RESPONSE %s-%d" % (response.status_code, number)
          #during this sleep time, check if the client socket connection is released or not on the API controller node.
          time.sleep(1000)
          print “Thread %d complete" % number
      except requests.exceptions.RequestException as ex:
          print “Exception occurred %d-%s" % (number, str(ex))

  if __name__ == '__main__':
      processes = []
      for number in range(40):
          p = Process(target=request, args=(number,))
          p.start()
          processes.append(p)
      for p in processes:
          p.join()

  ================================================================================================

  Presently, the wsgi server allows persist connections if you configure keepalive to True which is default.
  In order to close the client socket connection explicitly after the response is sent and read successfully by the client, you simply have to set keepalive to False when you create a wsgi server.

  Additional information: By default eventlet passes “Connection: keepalive” if keepalive is set to True when a response is sent to the client. But it doesn’t have capability to set the timeout and max parameter.
  For example.
  Keep-Alive: timeout=10, max=5

  Note: After we have disabled keepalive in all the OpenStack API
  service using wsgi library, then it might impact all existing
  applications built with the assumptions that OpenStack API services
  uses persistent connections. They might need to modify their
  applications if reconnection logic is not in place and also they might
  experience the performance has slowed down as it will need to
  reestablish the http connection for every request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1361360/+subscriptions