[infra][dev] Zuul promote pipeline for container publishing
James E. Blair
corvus at inaugust.com
Wed Jan 23 23:28:55 UTC 2019
We recently added a new pipeline to OpenStack's Zuul, and three new jobs
and roles to the zuul-jobs repo to support container publishing. If
your project publishes containers, or may do so in the future, read on.
The new pipeline is the "promote" pipeline. It runs jobs on changes
after they merge, but unlike the "post" pipeline, it does not run with
the resulting branch state after the merge -- it runs in the context of
the change. This means that it's not suitable for building release
artifacts, but that's not its purpose.
Its purpose is to promote artifacts previously built in the gate
pipeline to production. That is safe to do because anything built in
the gate pipeline is built with the full future state of in-flight
changes to all related projects and branches (even if it's built before
they land). We build a lot of artifacts in the gate pipeline now, and
then throw them away only to build them again in post. The promote
pipeline is an attempt to avoid that waste and instead preserve
artifacts built in gate and directly publish them -- but only if and
when the changes for which they were built are merged.
The first functional use of this system is in building container images
and publishing them on Docker Hub. We have begun using this in
openstack-infra to build container images of third-party applications,
but this will work just as well and as easily for building images of
applications in our infrastructure (for example, we are also starting to
use this to publish Zuul images).
To implement this, we wrote three jobs which are available now from
zuul-jobs (and therefore can be used in any project or job in the
* build-docker-image 
* upload-docker-image 
* promote-docker-image 
The first job should run in the check pipeline and simply runs "docker
build" on the arguments passed to it. It is used to verify that the
The second job should run in the gate pipeline. It uploads the image to
Docker Hub into the final repository location, but it only does so with
a single tag with the form "change_123456" where '123456' is the Gerrit
change number for the change under consideration. This will mean that
docker pulls won't accidentally fetch the image yet. But the image is
staged in Docker Hub and is ready to be promoted if the change passes
all its tests and merges.
The third job runs in the promote pipeline. This job, unlike the
others, only performs Docker Hub API calls and requires no build
resources. Therefore it is a zero-node job in Zuul -- a job which only
runs on the Zuul executor. This job finds a previously uploaded image
with a "change_" tag, and re-tags it with any specified tags ("latest"
by default, but you can also tag version numbers or anything else). It
also cleans up unused "change_" tags so they don't clutter up the
The nice thing about zero-node jobs is that they start and finish very
quickly (since they don't have to deal with resource allocation and
contention). With this system, an image will be fully published within
about a minute after the change merges.
To use this system, you will need to create (if you haven't done so
already) a credential which is permitted to upload to an organization on
Docker Hub, and you will need to add a secret  to Zuul for it.
A complete example .zuul.yaml might look like this:
description: Build Docker images.
- context: .
description: Build Docker images and upload to Docker Hub.
- name: docker_credentials
description: Promote previously uploaded Docker images.
- name: docker_credentials
If you find that you need jobs which behave slightly differently, you
may be able to inherit from these jobs and add pre or post playbooks.
Or if you need something particularly complicated, you can re-implement
the jobs using the underlying roles in zuul-jobs, which have the same
names an the jobs which invoke them.
More information about the openstack-discuss