[openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

Mike Bayer mbayer at redhat.com
Wed Jan 6 22:02:49 UTC 2016



On 01/06/2016 09:11 AM, Roman Podoliaka wrote:
> Hi Mike,
> 
> Thank you for this brilliant analysis! We've been seeing such timeout
> errors in downstream periodically and this is the first time someone
> has analysed the root cause thoroughly.
> 
> On Fri, Dec 18, 2015 at 10:33 PM, Mike Bayer <mbayer at redhat.com> wrote:
> 
>> But if we only have a super low number of greenlets and only a few dozen
>> workers, what happens if we have more than 240 requests come in at once,
>> aren't those connections going to get rejected?  No way!  eventlet's
>> networking system is better than that, those connection requests just
>> get queued up in any case, waiting for a greenlet to be available.  Play
>> with the script and its settings to see.
> 
> Right, it must be controlled by the backlog argument value here:
> 
> https://github.com/openstack/oslo.service/blob/master/oslo_service/wsgi.py#L80

oh wow, totally missed that!  But, how does backlog here interact with
multiple processes?   E.g. all workers are saturated, it will place a
waiting connection onto a random greenlet which then has to wait?  It
would be better if the "backlog" were pushed up to the parent process,
not sure if that's possible?





More information about the OpenStack-dev mailing list