Open Stack

Great, yeah we have also seen these issues with nova-api with keystonemiddle in newton and ocata.

Thanks for the heads up as I was going to start digging deeper.

Cheers,
Sam




> On 31 Jul 2018, at 10:09 am, iain MacDonnell <iain.macdonnell at oracle.com> wrote:
> 
> 
> Following up on my own question, in case it's useful to others....
> 
> Turns out that keystonemiddleware uses eventlet, and, by default, creates a connection to memcached from each green thread (and doesn't clean them up), and the green threads are essentially unlimited.
> 
> There is a solution for this, which implements a shared connection pool. It's enabled via the keystone_authtoken.memcache_use_advanced_pool config option.
> 
> Unfortunately it was broken in a few different ways (I guess this means that no one is using it?)
> 
> I've worked with the keystone devs, and we were able to get a fix (in keystonemiddleware) in just in time for the Rocky release. Related fixes have also been backported to Queens (for the next update), and a couple needed for Pike are pending completion.
> 
> With this in place, so-far I have not seen more than one connection to memcached for each neutron-api worker process, and everything seems to be working well.
> 
> Some relevant changes:
> 
> master:
> 
> https://review.openstack.org/#/c/583695/
> 
> 
> Queens:
> 
> https://review.openstack.org/#/c/583698/
> https://review.openstack.org/#/c/583684/
> 
> 
> Pike:
> 
> https://review.openstack.org/#/c/583699/
> https://review.openstack.org/#/c/583835/
> 
> 
> I do wonder how others are managing memcached connections for larger deployments...
> 
>    ~iain
> 
> 
> 
> On 06/26/2018 12:59 PM, iain MacDonnell wrote:
>> In diagnosing a situation where a Pike deployment was intermittently slower (in general), I discovered that it was (sometimes) exceeding memcached's maximum connection limit, which is set to 4096.
>> Looking closer, ~2750 of the connections are from 8 neutron-server process. neutron-server is configured with 8 API workers, and those 8 processes have a combined total of ~2750 connections to memcached:
>> # lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
>>     245 2611
>>     306 2612
>>     228 2613
>>     406 2614
>>     407 2615
>>     385 2616
>>     369 2617
>>     398 2618
>> #
>> There doesn't seem to be much turnover - comparing samples of the connections (incl. source port) 15 mins apart, two were dropped, and one new one added.
>> In neutron.conf, keystone_authtoken.memcached_servers is configured, but nothing else pertaining to caching, so keystone_authtoken.memcache_pool_maxsize should default to 10.
>> Am I misunderstanding something, or shouldn't I see a maximum of 10 connections from each of the neutron-server API workers, with this configuration?
>> Any known issues, or pointers to what I'm missing?
>> TIA,
>>     ~iain
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators