Hello,

python-memcached badly handles connections during a flush on reconnect and so connections can grow up exponentially [1].

I don't know if it is the same issue that you faced but it could be a track to follow.

On oslo.cache a fix has been submitted but it is not yet merged [2].

[1] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[2] https://review.opendev.org/#/c/742193/

Le ven. 11 sept. 2020 à 23:29, Tony Liu <tonyliu0592@hotmail.com> a écrit :
Hi,

Is there any guidance or experiences to estimate the number
of memcached connections?

Here is memcached connection on one of the 3 controllers.
Connection number is the total established connections to
all 3 memcached nodes.

Node 1:
10 Keystone workers have 62 connections.
11 Nova API workers have 37 connections.
6 Neutron server works have 4304 connections.
1 memcached has 4973 connections.

Node 2:
10 Keystone workers have 62 connections.
11 Nova API workers have 30 connections.
6 Neutron server works have 3703 connections.
1 memcached has 4973 connections.

Node 3:
10 Keystone workers have 54 connections.
11 Nova API workers have 15 connections.
6 Neutron server works have 6541 connections.
1 memcached has 4973 connections.

Before I increase the connection limit for memcached, I'd
like to understand if all the above is expected?

How Neutron server and memcached take so many connections?

Any elaboration is appreciated.

BTW, the problem leading me here is memcached connection timeout,
which results all services depending on memcached stop working
properly.


Thanks!
Tony




--
Hervé Beraud
Senior Software Engineer
Red Hat - Openstack Oslo
irc: hberaud
-----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
v6rDpkeNksZ9fFSyoY2o
=ECSj
-----END PGP SIGNATURE-----