Open Stack

Hi,

There are some changes to Zuul's behavior that will be arriving
beginning this Friday, July 19th.  Here's what you need to know:

* Zuul will be more aggressive at dequeuing changes in the gate
  pipeline when there are merge conflicts.

* Zuul will prioritize jobs for the gate pipeline ahead of all others,
  followed by the post-commit, and then check pipelines

Merge conflicts:

Currently our projects move so quickly that something like one in every
6 changes that is approved has a merge conflict.  Generally this means
that Zuul can start testing only about 5 changes before it runs into a
change that can't be merged with the changes ahead of it.  It then waits
until that change reaches the head of the queue, just in case a change
ahead fails to merge, and the merge conflict becomes moot.  That doesn't
usually happen, so the result is that Zuul does not test as many changes
as it could in parallel, and developers have to wait quite a while to
learn that their change did not merge simply because of a merge
conflict.

At the Havana summit, we decided it would be better for Zuul to
immediately dequeue changes in the gate queue that have a merge conflict
with the changes ahead of it.  This means that you'll immediately learn
that there is a conflict with other changes that have been approved
ahead of yours, and Zuul can test and merge more changes in parallel
(lowering overall time-to-merge for all changes).  If this happens, Zuul
will leave a message in Gerrit indicating which changes were enqueued
ahead which may have caused the merge conflict:

  This change was unable to be automatically merged with the current
  state of the repository and the following changes which were enqueued
  ahead of it: 1234, 5678. Please rebase your change and upload a new
  patchset.

If you see a message like that, you can either wait for those changes to
merge (or not) and rebase on the tip, or you can rebase your change on
the changes currently in the queue to be certain it can be merged.

Job precedence:

We recently changed Zuul to use Gearman as a job dispatching system.
Zuul now submits job requests to Gearman, and Jenkins (using the
gearman-plugin written by Khai Do) picks up those jobs and runs them.
This gives us a lot of new flexibility of which we're going to start
taking advantage in the future.  One of the first new features this
system gives us is the ability to set the precedence of jobs based on
our own criteria.  We plan on giving jobs in the gate pipeline the
highest precedence, so that as long as there are outstanding gate jobs,
they will be run as soon as resources are available.  Next, we'll
allocate resources to post-merge jobs (such as tarball and docs builds
and uploads), and finally to the check pipeline.

We have a number of other cool features planned, such as reporting on
build progress directly on the Zuul status page, and perhaps most
importantly, Gearman lets us scale out our Jenkins setup to multiple
masters, and even run jobs that aren't in Jenkins at all.

As usual, the Infrastructure Team will be available in #openstack-infra
to provide assistance as these new features are implemented.

-Jim