<div dir="ltr"><div>James,</div><div><br></div>Awesome! Amazing! You guys rock!=)<div><br></div><div>Best regards,</div><div>Boris Pavlovic </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 11, 2015 at 1:26 AM, James E. Blair <span dir="ltr"><<a href="mailto:corvus@inaugust.com" target="_blank">corvus@inaugust.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We have added support for cross-repo dependencies (CRD) in Zuul. The<br>
important bits:<br>
<br>
* To use them, include "Depends-On: <gerrit-change-id>" in the footer of<br>
your commit message. Use the full Change-ID ('I' + 40 characters).<br>
<br>
* These are one-way dependencies only -- do not create a cycle.<br>
<br>
* This is what all the grey dots and lines are in the check pipeline.<br>
<br>
Cross-Repo Dependencies Explained<br>
=================================<br>
<br>
There are two behaviors that might go by the name "cross-repo<br>
dependencies". We call them one-way and multi-way.<br>
<br>
Multi-way CRD allow for bidirectional links between changes. For<br>
instance, A depends on B and B depends on A. The theory there is that<br>
both would be tested together and merged as a unit. This is _not_ what<br>
we have implemented in Zuul. Discussions over the past two years have<br>
revealed that this type of behavior could cause problems for continuous<br>
deploments as it means that two components must be upgraded<br>
simultaneously. Not supporting this behavior is a choice we have made.<br>
<br>
One-way CRD behaves like a directed acyclic graph (DAG), like git<br>
itself, to indicate a one-way dependency relationship between changes in<br>
different git repos. Change A may depend on B, but B may not depend on<br>
A. This is what we have implemented in Zuul.<br>
<br>
Gate Pipeline<br>
=============<br>
<br>
When Zuul sees CRD changes, it serializes them in the usual manner when<br>
enqueuing them into a pipeline. This means that if change A depends on<br>
B, then when they are added to the gate pipeline, B will appear first<br>
and A will follow. If tests for B fail, both B and A will be removed<br>
from the pipeline, and it will not be possible for A to merge until B<br>
does.<br>
<br>
Note that if changes with CRD do not share a change queue (such as the<br>
"integrated gate" then Zuul is unable to enqueue them together, and the<br>
first will be required to merge before the second is enqueued.<br>
<br>
Check Pipeline<br>
==============<br>
<br>
When changes are enqueued into the check pipeline, all of the related<br>
dependencies (both normal git-dependencies that come from parent commits<br>
as well as CRD changes) appear in a dependency graph, as in gate. This<br>
means that even in the check pipeline, your change will be tested with<br>
its dependency. So changes that were previously unable to be fully<br>
tested until a related change landed in a different repo may now be<br>
tested together from the start.<br>
<br>
All of the changes are still independent (so you will note that the<br>
whole pipeline does not share a graph as in gate), but for each change<br>
tested, all of its dependencies are visually connected to it, and they<br>
are used to construct the git references that Zuul uses when testing.<br>
When looking at this graph on the status page, you will note that the<br>
dependencies show up as grey dots, while the actual change tested shows<br>
up as red or green. This is to indicate that the grey changes are only<br>
there to establish dependencies. Even if one of the dependencies is<br>
also being tested, it will show up as a grey dot when used as a<br>
dependency, but separately and additionally will appear as its own red<br>
or green dot for its test.<br>
<br>
(If you don't see grey dots on the status page, reload the page to get<br>
the latest version.)<br>
<br>
Multiple Changes<br>
================<br>
<br>
A Gerrit change ID may refer to multiple changes (on multiple branches<br>
of the same project, or even multiple projects). In these cases, Zuul<br>
will treat all of the changes with that change ID as dependencies. So<br>
if you say that a tempest change Depends-On a change ID that has changes<br>
in nova master and nova stable/juno, then when testing the tempest<br>
change, both nova changes will be applied, and when deciding whether the<br>
tempest change can merge, both changes must merge ahead of it.<br>
<br>
A change may depend on more than one Gerrit change ID as well. So it is<br>
possible for a change in tempest to depend on a change in devstack and a<br>
change in nova. Simply add more "Depends-On:" lines to the footer.<br>
<br>
Cycles<br>
======<br>
<br>
If a cycle is created by use of CRD, Zuul will abort its work very<br>
early. There will be no message in Gerrit and no changes that are part<br>
of the cycle will be enqueued into any pipeline. This is to protect<br>
Zuul from infinite loops. I hope that we can improve this to at least<br>
leave a message in Gerrit in the future. But in the meantime, please be<br>
cognizant of this and do not create dependency cycles with Depends-On<br>
lines.<br>
<br>
Examples<br>
========<br>
<br>
The following two infra changes have been tested together because of the<br>
Depends-On: line in the commit message of the first:<br>
<br>
<a href="https://review.openstack.org/#/c/152508/" target="_blank">https://review.openstack.org/#/c/152508/</a><br>
<a href="https://review.openstack.org/#/c/152504/" target="_blank">https://review.openstack.org/#/c/152504/</a><br>
<br>
In fact, you can see earlier test results failing until it was rechecked<br>
after CRD went into production (around 2015-02-10 15:20 UTC).<br>
<br>
This devstack change depended on a grenade change:<br>
<br>
<a href="https://review.openstack.org/#/c/154575/" target="_blank">https://review.openstack.org/#/c/154575/</a><br>
<a href="https://review.openstack.org/#/c/153702/" target="_blank">https://review.openstack.org/#/c/153702/</a><br>
<br>
And with CRD, was able to be tested in check together as well as being<br>
enqueued into the gate at the same time as its dependency.<br>
<br>
Note that in this example, the Gerrit change ID that 154575 depends on<br>
is actually associated with two changes on two separate branches. In<br>
cases such as this, Zuul treats all instances as dependencies (so both<br>
changes would be tested together, and both must merge before 154575<br>
may).<br>
<br>
Further Questions<br>
=================<br>
<br>
This is a fairly substantial new feature, and we may still have a bit to<br>
learn about it.<br>
<br>
If you have any further questions, feel free to ask here or in the<br>
#openstack-infra channel on Freenode.<br>
<br>
We will update the infra manual soon to reflect this change.<br>
<br>
-Jim<br>
<br>
_______________________________________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>
</blockquote></div><br></div>