<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 14 sept. 2020 à 18:09, Tony Liu <<a href="mailto:tonyliu0592@hotmail.com">tonyliu0592@hotmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Radosław pointed another bug<br>
<a href="https://bugs.launchpad.net/keystonemiddleware/+bug/1883659" rel="noreferrer" target="_blank">https://bugs.launchpad.net/keystonemiddleware/+bug/1883659</a><br>
referring to the same fix<br>
<a href="https://review.opendev.org/#/c/742193/" rel="noreferrer" target="_blank">https://review.opendev.org/#/c/742193/</a><br>
<br>
Regarding to the fix, The comment says "This flag is off by<br>
default for backwards compatibility.". But I see this flag is<br>
on by default in current code. That's how it causes issues.<br>
This fix changes the default value from on to off. It does break<br>
backwards compatibility. To keep Keystone working as the old way,<br>
along with this fix, this flag has to be explicitly set to true<br>
in keystone.conf. For neutron-server and nova-api, it's good to<br>
leave this flag off by default. Am I correct?<br>
<br></blockquote><div><br></div><div>Long short story as far as I correctly remember this topic.</div><div><br></div><div></div><div>Currently flush on reconnect is not an option and it is always triggered (in the corresponding scenario).</div><div><br></div><div>If we decide to introduce this new option `memcache_pool_flush_on_reconnect` we need to set this option to `True` as the default value to keep the backward compat.</div><div><br></div><div>If this option is set to `true` then flush on reconnect will be triggered all the time in the corresponding scenario. </div><div><br></div><div>Use `True` as default value was my first choice for these changes, and I think we need to give prior to backward compat for the first time and in a second time start by deprecating this behavior and turn this option to `False` as the default value if it helps to fix things.</div><div><br></div><div>Finally after some discussions `False` have been retained as default value (c.f comments on <a href="https://review.opendev.org/#/c/742193/">https://review.opendev.org/#/c/742193/</a>) which mean that flush on reconnect will not be executed and in this case I think we can say that backward compat is broken as this is not the current behavior.</div><div><br></div><div>AFAIK `flush_on_reconnect` have been added for Keystone and I think only Keystone really needs that but other people could confirm that.</div><div><br></div><div>If we decide to continue with `False` as the default value then neutron-server and nova-api could leave this default value as I don't think we need that (c.f my previous line).<br></div><div> </div><div>Finally, it could be worth to deep dive in the python-memcached side which is where the root cause is (the exponential connections) and to see how to address that.</div><div><br></div><div>Hope that helps you.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks!<br>
Tony<br>
> -----Original Message-----<br>
> From: Herve Beraud <<a href="mailto:hberaud@redhat.com" target="_blank">hberaud@redhat.com</a>><br>
> Sent: Monday, September 14, 2020 8:27 AM<br>
> To: Tony Liu <<a href="mailto:tonyliu0592@hotmail.com" target="_blank">tonyliu0592@hotmail.com</a>><br>
> Cc: openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> Subject: Re: memchached connections<br>
> <br>
> Hello,<br>
> <br>
> python-memcached badly handles connections during a flush on reconnect<br>
> and so connections can grow up exponentially [1].<br>
> <br>
> <br>
> I don't know if it is the same issue that you faced but it could be a<br>
> track to follow.<br>
> <br>
> On oslo.cache a fix has been submitted but it is not yet merged [2].<br>
> <br>
> <br>
> [1] <a href="https://bugs.launchpad.net/oslo.cache/+bug/1888394" rel="noreferrer" target="_blank">https://bugs.launchpad.net/oslo.cache/+bug/1888394</a><br>
> [2] <a href="https://review.opendev.org/#/c/742193/" rel="noreferrer" target="_blank">https://review.opendev.org/#/c/742193/</a><br>
> <br>
> Le ven. 11 sept. 2020 à 23:29, Tony Liu <<a href="mailto:tonyliu0592@hotmail.com" target="_blank">tonyliu0592@hotmail.com</a><br>
> <mailto:<a href="mailto:tonyliu0592@hotmail.com" target="_blank">tonyliu0592@hotmail.com</a>> > a écrit :<br>
> <br>
> <br>
> Hi,<br>
> <br>
> Is there any guidance or experiences to estimate the number<br>
> of memcached connections?<br>
> <br>
> Here is memcached connection on one of the 3 controllers.<br>
> Connection number is the total established connections to<br>
> all 3 memcached nodes.<br>
> <br>
> Node 1:<br>
> 10 Keystone workers have 62 connections.<br>
> 11 Nova API workers have 37 connections.<br>
> 6 Neutron server works have 4304 connections.<br>
> 1 memcached has 4973 connections.<br>
> <br>
> Node 2:<br>
> 10 Keystone workers have 62 connections.<br>
> 11 Nova API workers have 30 connections.<br>
> 6 Neutron server works have 3703 connections.<br>
> 1 memcached has 4973 connections.<br>
> <br>
> Node 3:<br>
> 10 Keystone workers have 54 connections.<br>
> 11 Nova API workers have 15 connections.<br>
> 6 Neutron server works have 6541 connections.<br>
> 1 memcached has 4973 connections.<br>
> <br>
> Before I increase the connection limit for memcached, I'd<br>
> like to understand if all the above is expected?<br>
> <br>
> How Neutron server and memcached take so many connections?<br>
> <br>
> Any elaboration is appreciated.<br>
> <br>
> BTW, the problem leading me here is memcached connection timeout,<br>
> which results all services depending on memcached stop working<br>
> properly.<br>
> <br>
> <br>
> Thanks!<br>
> Tony<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> --<br>
> <br>
> Hervé Beraud<br>
> Senior Software Engineer<br>
> <br>
> Red Hat - Openstack Oslo<br>
> irc: hberaud<br>
> -----BEGIN PGP SIGNATURE-----<br>
> <br>
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>
> v6rDpkeNksZ9fFSyoY2o<br>
> =ECSj<br>
> -----END PGP SIGNATURE-----<br>
> <br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer<br></div><div>Red Hat - Openstack Oslo</div><div>irc: hberaud</div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>