Open Stack

Hi,

Today we discussed a challenge around image tags that mostly boils
down to limitations in how overcloud_containers.yaml is constructed
and used.

TL;DR, we need a smart and easy way to work with the
overcloud_containers.yaml file (especially tags).

Let's highlight a few use cases that we need to work through:

#1. Building containers
  For building containers, all we really care is the name of the
images we need to build.
  Today, we install a trunk repository and then install
tripleo-common-containers from that trunk repository.
  We then mostly grep/sed/awk our way from overcloud_containers.yaml
to a clean list of images to build and then build those.
  Relatively okay with this but prone to things breaking -- a clean
way to get just the list of images out of there would be nice.

#2. Testing and promoting containers
  This comes right after use case #1 where we build containers in the pipeline.
  For those familiar with the CI pipeline to do promotions [1], this
would look a bit like this [2].

  In practice, this works basically the same way as we build, test and
promote tripleo-quickstart images.
  We pick a particular trunk repository hash and build containers for
that hash. These are then pushed with both the tags ":latest" and
":<dlrn_hash>".
  We're then supposed to test those containers in the pipeline but to
do that, we need to be pulling from :<dlrn_hash>, not :latest...
although they are in theory equivalent at that given time, this might
not always be true.
  So the testing job(s) need a way to easily customize/pull from that
particular hash instead of the hardcoded latest we have right now.

#3. Upstream gate jobs
  Ideally, gate jobs should never use ":latest". This is in trunk/dlrn
terms the equivalent of "/current/" or "/consistent/".
  They'd use something like ":latest-passed-ci" which would be the
proper equivalent of "/current-passed-ci/" or "/current-tripleo/".

  This brings an interesting challenge around how we currently add new
images to overcloud_containers.yaml (example [3]).
  It is expected that, when you add the image, the image is already
present on the registry because otherwise the container jobs will fail
since this new image cannot be pulled (example [4]).
  My understanding currently is that humans may build and push images
to the registry ahead of time so that this works.
  We can keep a similar approach if that's what we really want with
the new private registry, the job that builds container is made
generic exactly to be able to build just a specific set of image(s) if
we want.
  Here's the catch, though: this new container image will have the
":latest" tag, it will not have ":latest-passed-ci" because it hasn't
passed CI yet, it's being added just now.
  So how do we address this ?

  Note:
  We've already discussed that some containers need to pick up the
latest and the greatest from the "/current/" repository, either
because they are "direct" tripleo packages or if "Depends-On" is used.
  So far, the solution we seem to be going towards is to pick up the
containers from ":latest-passed-ci" and then more or less add a 'yum
update' layer to the images needing an update.
  This is the option that is in the best interest of time, we'd
otherwise be spending too much time building containers in jobs that
are already taking way too long to run.

#4. Test days
  When doing test days, we know to point testers to
/current-passed-ci/ as well as tested quickstart images.
  How can we make it easy for containers ? If the container list from
tripleo-common is hardcoded to latest, that won't work. If it's
hardcoded to :latest-passed-ci, it won't work for other use cases.
  Ideally this would be super easy for end users as well as developers
so that they can get started easily.

#5 Stable releases, users, operators & co
  The packaging workflow is not the same for trunk (dlrn out of git
source on trunk.rdoproject.oirg) and for stable releases (CentOS build
system out of released tarballs on mirror.centos.org).
  It's also going to be different for containers.
  For trunk, we'll be building containers with trunk repositories and
publishing them to a private registry analogous to
trunk.rdoproject.org repositories.
  For stable releases, while still hand-wavy and foggy, we seem to be
headed in the direction of the CentOS official registry which takes
Dockerfiles in pseudo dist-git repositories and builds/publishes the
containers through Jenkins jobs.
  This is sort of similar to how downstream would work if you replace
Jenkins by Brew/OSBS.

  So here, we want to use the overcloud_containers.yaml file to
"compile" Dockerfiles which will be shipped to git repositories and
then built by another process.
  These containers will be published somewhere that tripleo-common is
sort of expected to know ahead of time because users, customers and
developers need to be pulling from that "stable" source and from a
"stable" tag.

So... what do we do ?

[1]: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/
[2]: http://i.imgur.com/I26njTz.png
[3]: https://review.openstack.org/#/c/466305/
[4]: http://logs.openstack.org/51/467651/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/dc61632/logs/oooq/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-05-24_19_24_13

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]