Open Stack

Following up on my own question, in case it's useful to others....

Turns out that keystonemiddleware uses eventlet, and, by default, 
creates a connection to memcached from each green thread (and doesn't 
clean them up), and the green threads are essentially unlimited.

There is a solution for this, which implements a shared connection pool. 
It's enabled via the keystone_authtoken.memcache_use_advanced_pool 
config option.

Unfortunately it was broken in a few different ways (I guess this means 
that no one is using it?)

I've worked with the keystone devs, and we were able to get a fix (in 
keystonemiddleware) in just in time for the Rocky release. Related fixes 
have also been backported to Queens (for the next update), and a couple 
needed for Pike are pending completion.

With this in place, so-far I have not seen more than one connection to 
memcached for each neutron-api worker process, and everything seems to 
be working well.

Some relevant changes:

master:

https://review.openstack.org/#/c/583695/


Queens:

https://review.openstack.org/#/c/583698/
https://review.openstack.org/#/c/583684/


Pike:

https://review.openstack.org/#/c/583699/
https://review.openstack.org/#/c/583835/


I do wonder how others are managing memcached connections for larger 
deployments...

     ~iain



On 06/26/2018 12:59 PM, iain MacDonnell wrote:
> 
> In diagnosing a situation where a Pike deployment was intermittently 
> slower (in general), I discovered that it was (sometimes) exceeding 
> memcached's maximum connection limit, which is set to 4096.
> 
> Looking closer, ~2750 of the connections are from 8 neutron-server 
> process. neutron-server is configured with 8 API workers, and those 8 
> processes have a combined total of ~2750 connections to memcached:
> 
> # lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
>      245 2611
>      306 2612
>      228 2613
>      406 2614
>      407 2615
>      385 2616
>      369 2617
>      398 2618
> #
> 
> 
> There doesn't seem to be much turnover - comparing samples of the 
> connections (incl. source port) 15 mins apart, two were dropped, and one 
> new one added.
> 
> In neutron.conf, keystone_authtoken.memcached_servers is configured, but 
> nothing else pertaining to caching, so 
> keystone_authtoken.memcache_pool_maxsize should default to 10.
> 
> Am I misunderstanding something, or shouldn't I see a maximum of 10 
> connections from each of the neutron-server API workers, with this 
> configuration?
> 
> Any known issues, or pointers to what I'm missing?
> 
> TIA,
> 
>      ~iain