Open Stack

Hi,

We have added support for cross-repo dependencies (CRD) in Zuul.  The
important bits:

* To use them, include "Depends-On: <gerrit-change-id>" in the footer of
  your commit message.  Use the full Change-ID ('I' + 40 characters).

* These are one-way dependencies only -- do not create a cycle.

* This is what all the grey dots and lines are in the check pipeline.

Cross-Repo Dependencies Explained
=================================

There are two behaviors that might go by the name "cross-repo
dependencies".  We call them one-way and multi-way.

Multi-way CRD allow for bidirectional links between changes.  For
instance, A depends on B and B depends on A.  The theory there is that
both would be tested together and merged as a unit.  This is _not_ what
we have implemented in Zuul.  Discussions over the past two years have
revealed that this type of behavior could cause problems for continuous
deploments as it means that two components must be upgraded
simultaneously.  Not supporting this behavior is a choice we have made.

One-way CRD behaves like a directed acyclic graph (DAG), like git
itself, to indicate a one-way dependency relationship between changes in
different git repos.  Change A may depend on B, but B may not depend on
A.  This is what we have implemented in Zuul.

Gate Pipeline
=============

When Zuul sees CRD changes, it serializes them in the usual manner when
enqueuing them into a pipeline.  This means that if change A depends on
B, then when they are added to the gate pipeline, B will appear first
and A will follow.  If tests for B fail, both B and A will be removed
from the pipeline, and it will not be possible for A to merge until B
does.

Note that if changes with CRD do not share a change queue (such as the
"integrated gate" then Zuul is unable to enqueue them together, and the
first will be required to merge before the second is enqueued.

Check Pipeline
==============

When changes are enqueued into the check pipeline, all of the related
dependencies (both normal git-dependencies that come from parent commits
as well as CRD changes) appear in a dependency graph, as in gate.  This
means that even in the check pipeline, your change will be tested with
its dependency.  So changes that were previously unable to be fully
tested until a related change landed in a different repo may now be
tested together from the start.

All of the changes are still independent (so you will note that the
whole pipeline does not share a graph as in gate), but for each change
tested, all of its dependencies are visually connected to it, and they
are used to construct the git references that Zuul uses when testing.
When looking at this graph on the status page, you will note that the
dependencies show up as grey dots, while the actual change tested shows
up as red or green.  This is to indicate that the grey changes are only
there to establish dependencies.  Even if one of the dependencies is
also being tested, it will show up as a grey dot when used as a
dependency, but separately and additionally will appear as its own red
or green dot for its test.

(If you don't see grey dots on the status page, reload the page to get
the latest version.)

Multiple Changes
================

A Gerrit change ID may refer to multiple changes (on multiple branches
of the same project, or even multiple projects).  In these cases, Zuul
will treat all of the changes with that change ID as dependencies.  So
if you say that a tempest change Depends-On a change ID that has changes
in nova master and nova stable/juno, then when testing the tempest
change, both nova changes will be applied, and when deciding whether the
tempest change can merge, both changes must merge ahead of it.

A change may depend on more than one Gerrit change ID as well.  So it is
possible for a change in tempest to depend on a change in devstack and a
change in nova.  Simply add more "Depends-On:" lines to the footer.

Cycles
======

If a cycle is created by use of CRD, Zuul will abort its work very
early.  There will be no message in Gerrit and no changes that are part
of the cycle will be enqueued into any pipeline.  This is to protect
Zuul from infinite loops.  I hope that we can improve this to at least
leave a message in Gerrit in the future.  But in the meantime, please be
cognizant of this and do not create dependency cycles with Depends-On
lines.

Examples
========

The following two infra changes have been tested together because of the
Depends-On: line in the commit message of the first:

  https://review.openstack.org/#/c/152508/
  https://review.openstack.org/#/c/152504/

In fact, you can see earlier test results failing until it was rechecked
after CRD went into production (around 2015-02-10 15:20 UTC).

This devstack change depended on a grenade change:

  https://review.openstack.org/#/c/154575/
  https://review.openstack.org/#/c/153702/

And with CRD, was able to be tested in check together as well as being
enqueued into the gate at the same time as its dependency.

Note that in this example, the Gerrit change ID that 154575 depends on
is actually associated with two changes on two separate branches.  In
cases such as this, Zuul treats all instances as dependencies (so both
changes would be tested together, and both must merge before 154575
may).

Further Questions
=================

This is a fairly substantial new feature, and we may still have a bit to
learn about it.

If you have any further questions, feel free to ask here or in the
#openstack-infra channel on Freenode.

We will update the infra manual soon to reflect this change.

-Jim