Open Stack

Hi,

We have just changed the way Jenkins jobs are triggered for Gerrit
changes.  This is another in a series of changes we're rolling out
based on needs described at the design summit.

We had been using the Gerrit Trigger Plugin to launch check and gate
jobs, but we're starting to run up against some scalability limits.
As we process more changes, and the tests for those changes get
longer, Jenkins was started to build up significant queues at times.
If we were to start gating on Tempest, Jenkins may just end up running
continuously, with long delays between approval and merging of
changes.

We have switched to a new system called Zuul, which is a python daemon
that watches the Gerrit event stream and triggers Jenkins jobs.  Zuul
is extremely flexible, but its main advantage is that it will allow us
to perform speculative execution of Jenkins jobs.  The general idea is
that most Jenkins jobs succeed, and as long as that is the case, we
can execute gate tests with those jobs in parallel.  If they all
succeed, a large number of changes can be quickly merged.

Another change is that it is easy for us to construct job decision
trees and report that information meaningfully to Gerrit.  This way we
can avoid wasting time running unit tests on changes that don't even
merge successfully

One important difference is that the new system does not have
"retrigger" buttons in Jenkins.  If the gate tests fail with a false
negative, you'll need to leave another "Approved" vote in Gerrit.

We've been using Zuul for about a week to run testing for some of the
CI repositories.  We think it's ready to run at scale, but there are
bound to be a few hiccups, and we'll try to get them fixed as quickly
as possible.  I think being able to test and merge changes in parallel
will be a huge win, and we'll be able to do some cool things in the
future with the additional flexibility this brings.

For more details on the theory of operation, see:

  http://amo-probos.org/post/14

or the project itself:

  https://launchpad.net/zuul
  https://github.com/openstack-ci/zuul

or the configuration file:

  https://github.com/openstack/openstack-ci-puppet/blob/master/modules/openstack-ci-config/files/zuul/layout.yaml

As usual, everything is in git, and anyone can help contribute to this
effort.

-Jim