Open Stack

Hi,

We recently added a new pipeline to OpenStack's Zuul, and three new jobs
and roles to the zuul-jobs repo to support container publishing.  If
your project publishes containers, or may do so in the future, read on.

The new pipeline is the "promote" pipeline.  It runs jobs on changes
after they merge, but unlike the "post" pipeline, it does not run with
the resulting branch state after the merge -- it runs in the context of
the change.  This means that it's not suitable for building release
artifacts, but that's not its purpose.

Its purpose is to promote artifacts previously built in the gate
pipeline to production.  That is safe to do because anything built in
the gate pipeline is built with the full future state of in-flight
changes to all related projects and branches (even if it's built before
they land).  We build a lot of artifacts in the gate pipeline now, and
then throw them away only to build them again in post.  The promote
pipeline is an attempt to avoid that waste and instead preserve
artifacts built in gate and directly publish them -- but only if and
when the changes for which they were built are merged.

The first functional use of this system is in building container images
and publishing them on Docker Hub.  We have begun using this in
openstack-infra to build container images of third-party applications,
but this will work just as well and as easily for building images of
applications in our infrastructure (for example, we are also starting to
use this to publish Zuul images).

To implement this, we wrote three jobs which are available now from
zuul-jobs (and therefore can be used in any project or job in the
system):

* build-docker-image [1]
* upload-docker-image [2]
* promote-docker-image [3]

The first job should run in the check pipeline and simply runs "docker
build" on the arguments passed to it.  It is used to verify that the
build succeeds.

The second job should run in the gate pipeline.  It uploads the image to
Docker Hub into the final repository location, but it only does so with
a single tag with the form "change_123456" where '123456' is the Gerrit
change number for the change under consideration.  This will mean that
docker pulls won't accidentally fetch the image yet.  But the image is
staged in Docker Hub and is ready to be promoted if the change passes
all its tests and merges.

The third job runs in the promote pipeline.  This job, unlike the
others, only performs Docker Hub API calls and requires no build
resources.  Therefore it is a zero-node job in Zuul -- a job which only
runs on the Zuul executor.  This job finds a previously uploaded image
with a "change_" tag, and re-tags it with any specified tags ("latest"
by default, but you can also tag version numbers or anything else).  It
also cleans up unused "change_" tags so they don't clutter up the
repository.

The nice thing about zero-node jobs is that they start and finish very
quickly (since they don't have to deal with resource allocation and
contention).  With this system, an image will be fully published within
about a minute after the change merges.

To use this system, you will need to create (if you haven't done so
already) a credential which is permitted to upload to an organization on
Docker Hub, and you will need to add a secret [4] to Zuul for it.

A complete example .zuul.yaml might look like this:

    - secret:
        name: myproject-dockerhub
        data:
          username: dockerhubuser
          password: !encrypted/pkcs1-oaep
            - DFlbrDM5eUMptMGIVMXV1g455xOJLi92UYF08Z2/JlIGu3t6v052o9FKlVyj1ZmpXs5+2
              JTa5jHkLTvTsYs9fCaNcQc2nmViCyWNlbOMzjB17uiZOaYFNs1sMqZcUZbGEz7Y8ds6Qq
              NBXI10jWFPTah4QxUuBvUbT3vmjnUToCzexl5ZGhKgijcnROWfUsnlCdugpgoNIcPsUki
              zty5FotDihnrC8n8vIomVK6EClY38ty97pLrADzFDd+Cos/OUlvi2xooUhzx8Bn020rJA
              lqEU5v8LGXp5QkHx0MSDx6JY6KppJ/4p/yM+4By6l+A20zdcimxmgiNc9rMWPwDj7xsao
              m7NAZWmWqOO0Xkhgt6WOfugwgt9X46sgs2+yDEfbnI5ok8uRbAB/4FWj/KdpyXwhcf+O2
              wEfhxLwDbAoGONQPjb4YcZmCXtmR7Qe5t+n2jyczWXvrbaBDUQP5a+YtVNN/xhmQ7D740
              POlxv7bLxJAixzqaQ3d8Rz9ZEv6zzRuhWph32UQtZ1JxSNww+EvmXm2eEi2Q2z6pT1Cx/
              j2OrFyA2GL/UJOVb15VHKF6bgHPHWJtpjPFhqdcvBhVute4BWB+KPcWH+y+apHN1enK3H
              tNJO9iqm34nKwSuj5ExmFw50LtwR5/9FyRuRPq/vBL+8y82v8FDmeYsBeobn5M=
    
    - job:
        name: myproject-build-image
        parent: build-docker-image
        description: Build Docker images.
        vars: &image_vars
          docker_images:
            - context: .
              repository: myproject/imagename
    
    - job:
        name: myproject-upload-image
        parent: upload-docker-image
        description: Build Docker images and upload to Docker Hub.
        secrets:
          - name: docker_credentials
            secret: myproject-dockerhub
            pass-to-parent: true
        vars: *image_vars
    
    - job:
        name: myproject-promote-image
        parent: promote-docker-image
        description: Promote previously uploaded Docker images.
        secrets:
          - name: docker_credentials
            secret: myproject-dockerhub
            pass-to-parent: true
        vars: *image_vars
    
    - project:
        check:
          jobs:
            - myproject-build-image
        gate:
          jobs:
            - myproject-upload-image
        promote:
          jobs:
            - myproject-promote-image

If you find that you need jobs which behave slightly differently, you
may be able to inherit from these jobs and add pre or post playbooks.
Or if you need something particularly complicated, you can re-implement
the jobs using the underlying roles in zuul-jobs, which have the same
names an the jobs which invoke them.

[1] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-build-docker-image
[2] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-upload-docker-image
[3] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-promote-docker-image
[4] https://zuul-ci.org/docs/zuul/user/encryption.html