Open Stack

Reviewed:  https://review.openstack.org/164643
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=392dc228034bbd8968f4c65ddfce6343bff938ea
Submitter: Jenkins
Branch:    stable/icehouse

commit 392dc228034bbd8968f4c65ddfce6343bff938ea
Author: abhishekkekane <abhishek.kekane at nttdata.com>
Date:   Tue Oct 21 01:37:42 2014 -0700

    Eventlet green threads not released back to pool
    
    Presently, the wsgi server allows persist connections hence even after
    the response is sent to the client, it doesn't close the client socket
    connection.
    Because of this problem, the green thread is not released back to the pool.
    
    In order to close the client socket connection explicitly after the
    response is sent and read successfully by the client, you simply have to
    set keepalive to False when you create a wsgi server.
    
    Icehouse backport note: socket_timeout was dropped, it was introduced
    in 0.14[*] and Icehouse eventlet lower bound is 0.13
    
    [*] https://github.com/eventlet/eventlet/commit/7d4916f01462de09cb58853d9de2e85777c2ad5b
    
    Note: The required unit-tests are manually added to the below path,
    as new path for unit-tests is not present in stable/icehouse release.
    nova/tests/test_wsgi.py
    
    DocImpact:
    Added wsgi_keep_alive option (default=True).
    
    SecurityImpact
    
    Conflicts:
            nova/tests/unit/test_wsgi.py
    
    Closes-Bug: #1361360
    (cherry picked from commit 04d7a724fdf80db51e73f12c5b8c982db9310742)
    
    Change-Id: I3b14a37edbe4bdc5db31ff4f08f78e78b60077ff


** Changed in: glance/icehouse
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1361360

Title:
  Eventlet green threads not released back to the pool leading to
  choking of new requests

Status in Cinder:
  Fix Released
Status in Cinder icehouse series:
  Fix Released
Status in Cinder juno series:
  Fix Released
Status in OpenStack Image Registry and Delivery Service (Glance):
  Fix Released
Status in Glance icehouse series:
  Fix Committed
Status in Orchestration API (Heat):
  In Progress
Status in OpenStack Identity (Keystone):
  Fix Committed
Status in Keystone icehouse series:
  Confirmed
Status in Keystone juno series:
  Fix Committed
Status in OpenStack Neutron (virtual network service):
  Fix Released
Status in neutron icehouse series:
  Fix Released
Status in neutron juno series:
  Fix Committed
Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  In Progress
Status in OpenStack Security Advisories:
  Won't Fix
Status in OpenStack Data Processing (Sahara):
  Confirmed

Bug description:
  Currently reproduced  on Juno milestone 2. but this issue should be
  reproducible in all releases since its inception.

  It is possible to choke OpenStack API controller services using
  wsgi+eventlet library by simply not closing the client socket
  connection. Whenever a request is received by any OpenStack API
  service for example nova api service, eventlet library creates a green
  thread from the pool and starts processing the request. Even after the
  response is sent to the caller, the green thread is not returned back
  to the pool until the client socket connection is closed. This way,
  any malicious user can send many API requests to the API controller
  node and determine the wsgi pool size configured for the given service
  and then send those many requests to the service and after receiving
  the response, wait there infinitely doing nothing leading to
  disrupting services for other tenants. Even when service providers
  have enabled rate limiting feature, it is possible to choke the API
  services with a group (many tenants) attack.

  Following program illustrates choking of nova-api services (but this
  problem is omnipresent in all other OpenStack API Services using
  wsgi+eventlet)

  Note: I have explicitly set the wsi_default_pool_size default value to 10 in order to reproduce this problem in nova/wsgi.py.
  After you run the below program, you should try to invoke API
  ============================================================================================
  import time
  import requests
  from multiprocessing import Process

  def request(number):
     #Port is important here
     path = 'http://127.0.0.1:8774/servers'
      try:
          response = requests.get(path)
          print "RESPONSE %s-%d" % (response.status_code, number)
          #during this sleep time, check if the client socket connection is released or not on the API controller node.
          time.sleep(1000)
          print “Thread %d complete" % number
      except requests.exceptions.RequestException as ex:
          print “Exception occurred %d-%s" % (number, str(ex))

  if __name__ == '__main__':
      processes = []
      for number in range(40):
          p = Process(target=request, args=(number,))
          p.start()
          processes.append(p)
      for p in processes:
          p.join()

  ================================================================================================

  Presently, the wsgi server allows persist connections if you configure keepalive to True which is default.
  In order to close the client socket connection explicitly after the response is sent and read successfully by the client, you simply have to set keepalive to False when you create a wsgi server.

  Additional information: By default eventlet passes “Connection: keepalive” if keepalive is set to True when a response is sent to the client. But it doesn’t have capability to set the timeout and max parameter.
  For example.
  Keep-Alive: timeout=10, max=5

  Note: After we have disabled keepalive in all the OpenStack API
  service using wsgi library, then it might impact all existing
  applications built with the assumptions that OpenStack API services
  uses persistent connections. They might need to modify their
  applications if reconnection logic is not in place and also they might
  experience the performance has slowed down as it will need to
  reestablish the http connection for every request.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1361360/+subscriptions