Open Stack

Hi Anita,

Thanks for taking an action there. I wasn't expecting my work to affect infra that much. Apologize for that.
Please see my comments inline below.

> -----Original Message-----
> From: Anita Kuno [mailto:anteaya at anteaya.info]
> Sent: Thursday, June 4, 2015 8:15 PM
> To: Znoinski, Waldemar
> Cc: openstack-infra at lists.openstack.org
> Subject: Your Gerrit Account has been temporarily disabled
> 
> Hi:
> 
> We had an issue today with Gerrit being unable to broadcast stream events
> due to stuck threads. we were having issues with this in the past but they
> cleared up after we discovered that one of our third party ci operators had a
> script running to keep ssh connections open. Today was the first time since
> we had the third party ci operator turn off there script that we had an issue.
> 
> Jeremy looked and discovered that there were a number of threads
> associated with your personal gerrit account and in an attempt to get things
> moving again disabled your personal gerrit account.

I'd like to know more about what you saw and/or what was causing (or you think was) the problem. 
* Was it too many connections spawn in a given amount of time?
* Were the connections long lasting (possible lack of closing the connections)?
* Was the command inside the ssh session not finishing/hanging (or long running) ?
What I see my side for last 24h period is ~10 connections to review.openstack.org which were hanging and not closed my side, yet not doing anything as far as I can tell. From your description of the problem that may be it - Gerrit threads consumed unnecessarily. If you have connection details of the problematic ssh sessions (source port at least) it would be great.
As I understand listening of Gerrit event stream is not causing the issue but the second part (to run 'gerrit query' over ssh) is - correct me if I'm wrong.

> 
> Were you running any script or automated tool to do with ssh connections
> out of your personal gerrit account? Can you think of anything unusual that
> might be associated with your account? We are interested to hear from you.

Indeed I was testing a check (python script using pygerrit) that would listen on Gerrit stream events and run Gerrit query to get more details about the change (to get comments on current patchset in particular).
The script runs one "gerrit query" per Gerrit event given:
* event is for project Openstack/neutron or Openstack/nova 
* event is from username 'jenkins' or 'intel-networking-ci'
* event is of type 'Comment Added'
* event applies to a Gerrit change
"gerrit query" is not run unless all 4 points above are True so I don't think there will be many of them (2 per patchset per project)

The purpose of that check is to compare results of OS Infra Jenkins with our own CI job. 
This check would enable us to see when there may be a problem with the infrastructure piece of our CI, i.e.: if Jenkins gave SUCCESS and our own CI had FAILED - we would then investigate any problems with our CI.

> 
> Once we get this cleared up we will re-enable your account. It is entirely
> possible you did something you consider harmless and gerrit responded
> unexpectedly. We appreciate your help.
> 
> Also as a note, if testing anything that pertains to a ci system, that activity is
> best done via a ci account, not a personal gerrit account.

I do agree and I will be using CI account for that once we have the situation clear and problem sorted including a possible different approach.

> 
> Thanks,
> Anita.
> 
> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-
> infra.2015-06-04.log.html#t2015-06-04T18:24:26
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.