I believe you are hitting [1].
We have recently worked around that in Kolla-Ansible [2].
We'd like to get clarifications on whether the workaround is the right choice but it seems to work. :-)
[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659 [2] https://review.opendev.org/746966
-yoctozepto
On Sat, Sep 12, 2020 at 4:34 AM Tony Liu tonyliu0592@hotmail.com wrote:
I restarted neutron-server on all 3 nodes. Those memcached connections from neutron-server are gone. Everything is back to normal. It seems like that memcached connections are not closed properly in neutron-server. The connections pile up over time. Is there any know issue related? Could any Neutron experts comment here?
Thanks! Tony
-----Original Message----- From: Tony Liu tonyliu0592@hotmail.com Sent: Friday, September 11, 2020 2:26 PM To: openstack-discuss openstack-discuss@lists.openstack.org Subject: memchached connections
Hi,
Is there any guidance or experiences to estimate the number of memcached connections?
Here is memcached connection on one of the 3 controllers. Connection number is the total established connections to all 3 memcached nodes.
Node 1: 10 Keystone workers have 62 connections. 11 Nova API workers have 37 connections. 6 Neutron server works have 4304 connections. 1 memcached has 4973 connections.
Node 2: 10 Keystone workers have 62 connections. 11 Nova API workers have 30 connections. 6 Neutron server works have 3703 connections. 1 memcached has 4973 connections.
Node 3: 10 Keystone workers have 54 connections. 11 Nova API workers have 15 connections. 6 Neutron server works have 6541 connections. 1 memcached has 4973 connections.
Before I increase the connection limit for memcached, I'd like to understand if all the above is expected?
How Neutron server and memcached take so many connections?
Any elaboration is appreciated.
BTW, the problem leading me here is memcached connection timeout, which results all services depending on memcached stop working properly.
Thanks! Tony