Open Stack

Hi,

As mentioned in the thread "Jenkins and transient failures", we've had
an unusually high number of transient failures in Jenkins lately.  We've
done several things in response to that:

1) Monty identified a problem with our pypi mirror which was the cause
of many of the errors, and corrected it.

2) Monty is continuing to work on the single dependency list which
should allow us to switch to using our local pypi mirror exclusively,
further reducing transient network errors, as well as significantly
speeding up test run time.

3) Several transient errors were caused by failed fetches from Gerrit.
While consulting with the Gerrit authors about tuning, they discovered a
bug in Gerrit where a 5 minute timeout was being interpreted as a 5
millisecond timeout.  I have updated our gerrit configuration to work
around that.

4) Clark Boylan implemented automatic retrying for the git fetches that
we use with Jenkins.


I hope that we'll get to the point where we have almost no transient
network errors when testing, but we know it will never be perfect, so at
the CI meeting we discussed how best to implement retriggering with
Zuul.  Clark added a comment filter that will retrigger Jenkins if you
leave a comment that matches a regex.

We currently run two kinds of jobs in Jenkins, the check job and the
gate job.  The check jobs run immediately when a patchset is uploaded
and vote +/-1.  The gate jobs run on approval, queue up across all
projects and vote +/-2 (if they fail, jobs behind them in the gate
pipeline may need to run again).


To retrigger the initial Jenkins check job, just leave a comment on the
review in Gerrit with only the text "recheck".

To retrigger the Jenkins merge gate job, leave a comment with only the
text "reverify", or if you are a core reviewer, just leave another
"Approved" vote.  (Don't leave a "reverify" comment if the change hasn't
been approved yet, it still won't be merged and will slow Jenkins down.)

-Jim