Open Stack

Hello,

I believe I have tracked down the issue. When setting up our Fibre Channel
CI, we created a script to listen to the gerrit event stream that was
using the Œnetapp-ci¹ account. This script had a code path where the
paramiko ssh connection would not be closed and this has been resolved.
Evidence that this was the culprit is that the script had been running
without interruption since April 29th and that is consistent with the
Gerrit connections from this account we see here:
http://paste.openstack.org/show/215173/

I apologize for the inconvenience and appreciate all the help in
#openstack-infra. I think it is safe for the account to be reenabled.

Thanks,

-Alex


On 5/5/15, 8:50 PM, "James E. Blair" <corvus at inaugust.com> wrote:

>Hi,
>
>We've been tracking a bug in Gerrit recently where all of the threads
>tasked with servicing the stream-events command eventually get stuck.
>This causes all of the CI systems, including OpenStack's, to stop
>responding to events until the server is manually restarted.
>
>We recently found that had happened with connections from the netapp-ci
>account.  I believe that Gerrit should be more resilient to these kinds
>of errors, however, due to the severe impact to the project when this
>happens, I have disabled the netapp-ci account until we find a solution
>to the problem.
>
>Note that the Gerrit upgrade scheduled for Saturday May 9 will bring a
>new SSH server with it, and may have an impact on this issue.
>
>If the netapp-ci operators have a moment to chat with us in
>#openstack-infra on Freenode that would probably be the best way to work
>on a plan to debug the problem further.
>
>Thanks, and sorry for the inconvenience,
>
>Jim
>
>_______________________________________________
>Third-party-announce mailing list
>Third-party-announce at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce