[openstack-dev] [all] Rollout of Zuul v3 at the PTG

Monty Taylor mordred at inaugust.com
Wed Aug 2 16:21:11 UTC 2017

Hey everybody!

The time has come at long last, we're actually aiming at a concrete date 
for rolling Zuul v3 live. \o/

tl;dr - Assuming things go well we want to do it at the PTG

We couldn't be more excited - there are a giant pile of feature requests 
from folks that we've been responding with "yes, as soon as Zuul v3 is 
here you can do that" - and that's not any more fun for us as it is for you.

If you have no idea what Zuul v3 is - there is a section at the end of 
the email just for you.

The Plan

Zuul v3 itself works great - we're using it to run tests on Zuul itself. 
The biggest bulk of work at the moment is ... job content.

We know that we can do a one-time migration of job content - because we 
do it every day for our current jobs (they are translated from 
jenkins-job-builder to ansible on the fly in Zuul 2.5 today) However, 
that translated content is ugly - and Zuul v3 allows us to make a bunch 
of things nicer. We'd like to have as many of our jobs translated to 
'nice' versions for the cutover as possible so that as people copy-pasta 
jobs they've got good content to work from.

We're currently working on writing v3-native versions of the most common 
jobs. This includes the tox-based unittest jobs and common things like 
tarball creation, pypi and docs publication. Next up is getting 
devstack-gate jobs done. Between tox and devstack-gate - we should have 
the majority of our jobs covered, so any outliers we couldn't 
auto-translate into something nice and new will have good examples to 
work from for followup cleaning.

devstack-gate question

There is an open question for devstack-gate. Namely - given the 
mechanism of how it works with defining a bunch of environment variables 
and then defining a shell function - it's currently unknown if we'll be 
able to fully auto-translate all of the existing project-specific 
special devstack-gate jobs, or whether we'll need to just migrate them 
all to things that look pretty similar to how they look now and have a 
few clear examples people can follow as they feel like updating things.

We should know more about that soon as we're working devstack-gate jobs 
this week.

Go Live

On the Saturday and Sunday immediately preceeding the PTG, we will run a 
few trial cutovers. That is, we'll do a short zuul downtime, run job 
migration scripts, turn zuul v3 on, run for a little bit to see how 
things go, then turn it off and turn v2 back on. This will let us 
double-check the migration during a time that's likely slower than 
normal for any last minute dead-in-the-water issues.

First thing Monday morning of the PTG, we'll throw the switch for real. 
We'll be around all week in a sort of "open office hours" form to help 
folks if there are any questions, if there are any job execution edge 
cases that need to be worked through - or if people are now excited by 
all the new features and have questions about how to make use of them.

We'll essentially be planning on our PTG to be all about making sure any 
questions any of you have can be fielded with plenty of bandwidth.

It's possible we won't be ready. We're pretty optimistic, but this is a 
big move, and something unforseen could come up between now and then. 
We're not going to throw the switch without being confident that things 
will go well just to meet a date. If we decide to cancel the cutover, 
we'll send an announcement to the list

Between Now and Then

By and large the leadup to this should have very little impact on folks.

There are some jobs that use scripts that are pre-baked into images that 
come from the jenkins/scripts directory in project-config. Those will no 
longer be used in Zuul v3 and will be migrated into Zuul v3 job 
definitions. Those scripts are currently under a soft-freeze, meaning 
that we'd prefer to not make any changes to them - but if a change is 
needed we need to ensure it gets forward-ported to the zuulv3 version. 
Closer to the PTG we'll put those scripts under a hard-freeze. They're 
all tricky to deal with anyway because they get baked in to images, so 
we're not expecting a ton of impact from that.

We may need to freeze all of project-config in the days leading up to 
the cutover, but since those are the days leading up to the PTG it's 
unlikely that should be a super active time for that repo.

Third Party CI

Any Third Party CI that is currently using Zuul should continue using 
Zuul v2 as you are today. Once we go live for OpenStack we'll be in a 
position to start discussing upgrade plans for 3rd Party operators. The 
goal is to migrate all of you - but we don't want folks to start 
tackling that until we're happy we've got automation and documentation 
worked out. After the PTG will be a great time to start discussing what 
that plan wants to look like.

Wait - what the heck is Zuul v3 and why are we doing this?

We've started a documentation page on this topic in the Infra Manual. 
It is short right now, but as we get closer to the cutover, we will 
continue to update this page with information about the migration:


Zuul v3 is the new version of Zuul that incorporates a ton of learning 
about pain points from the last several years. It adds in-repo config, 
native multi-node testing, ansible-based job definitions, full DAG job 
dependencies and support for integration with systems OpenStack doesn't 
otherwise use such as GitHub.

Zuul v3 is designed to be run by not-Infra. We're happy that Zuul has 
been useful for other people over the past few years, especially our 
Third-Party CI community - but up until v3 the OpenStack project was 
always consideration #1 and other users were best-effort.

With v3 we formally consider users who are not the OpenStack Project to 
be first-class users. We have recently added 3 core reviewers to Zuul 
who are not involved with running OpenStack's Zuul. (one runs a Zuul v3 
at BMW and two run a Zuul v3 for the IBM BonnyCI project)

We have a bunch of work teed up for post PTG rollout to add more support 
for a wider array of usecases. The OpNFV Infra team has an important 
feature request that Tristan from the RH Software Factory team has 
started implementing. We have planned support for container execution, 
both on a single-container and a COE (kubernetes/mesos) level, as well 
as additional triggers/reporters such as FedMsg as used by Fedora and 
Debian projects.

More information on the background of Zuul v3 can be found at:




Docs for Zuul v3 itself can currently be found at:


As usual, you can find us in #openstack-infra and in #zuul.


More information about the OpenStack-dev mailing list