Open Stack

On Mon, Oct 16, 2017, at 08:00 PM, guy.kim at itri.org.tw wrote:
> Dear everyone,
> 
> I am Guy, one of the operator for our third-party CI.
> 
> Recently, we are facing an issue in our CI. Indeed, our CI keeps running
> and reports the results to Gerrit. But there are still some remaining
> connections to Gerrit that are not closed. OpenStack Infra said that it
> reaches more than 100 connections (limit number of allowed connections).
> So when the connection is over 100, our CI cannot connect to Gerrit
> because the IP is blocked due to the high number of remaining
> connections.
> 
> We are still checking our servers if one of them keeps dropping SSH
> connections to Gerrit at the port 29418. So far, we haven’t found any
> servers trying to connect to Gerrit.
> 
> Besides, our zuul version is 2.5.2. This version has already includes the
> fix for https://storyboard.openstack.org/#!/story/2000923.
> 
> Is anyone has faced this issue before? If yes, how did you fix it?
> 
> Thank you in advance for your answers.
> Best regards,
> Guy KIM

First thing that comes to mind is perhaps some firewall is timing out
the connections but only letting the client know, so the connection
remains open on the Gerrit server. Another possibility is if you have
many Gerrit ssh clients behind a single address via NAT that you would
potentially see this problem if enough of them were connected at once.

It might help if you got on Freenode's IRC servers and joined
#openstack-infra to interactively debug this with the Gerrit admins.

Hope this helps,
Clark