Open Stack

On Sat, May 27, 2017 at 7:07 AM, David Moreau Simard <dms at redhat.com> wrote:

> Hi,
>
> Today we discussed a challenge around image tags that mostly boils
> down to limitations in how overcloud_containers.yaml is constructed
> and used.
>
> TL;DR, we need a smart and easy way to work with the
> overcloud_containers.yaml file (especially tags).
>
> Let's highlight a few use cases that we need to work through:
>
> #1. Building containers
>   For building containers, all we really care is the name of the
> images we need to build.
>   Today, we install a trunk repository and then install
> tripleo-common-containers from that trunk repository.
>   We then mostly grep/sed/awk our way from overcloud_containers.yaml
> to a clean list of images to build and then build those.
>   Relatively okay with this but prone to things breaking -- a clean
> way to get just the list of images out of there would be nice.
>
>
The command "openstack overcloud container image build" also has some
string matching logic, but then invokes kolla-build directly.

Can I suggest that we add a --list-images option to this command so that it
just returns a list of images for other image building tools to consume?


> #2. Testing and promoting containers
>   This comes right after use case #1 where we build containers in the
> pipeline.
>   For those familiar with the CI pipeline to do promotions [1], this
> would look a bit like this [2].
>
>   In practice, this works basically the same way as we build, test and
> promote tripleo-quickstart images.
>   We pick a particular trunk repository hash and build containers for
> that hash. These are then pushed with both the tags ":latest" and
> ":<dlrn_hash>".
>   We're then supposed to test those containers in the pipeline but to
> do that, we need to be pulling from :<dlrn_hash>, not :latest...
> although they are in theory equivalent at that given time, this might
> not always be true.
>   So the testing job(s) need a way to easily customize/pull from that
> particular hash instead of the hardcoded latest we have right now.
>
>
I would like to see another "openstack overcloud container image ..."
command which is pointed at an image registry and a
canonical overcloud_containers.yaml file, then generates
another overcloud_containers.yaml (and heat environment file) which
contains the proper latest <dlrn_hash> tags. This tool could work too for
stable version-style tags.

How about "openstack overcloud container image discover"?

This would be easier to implement if the canonical
overcloud_containers.yaml file was a template rather than a file with
hard-coded namespace and tags.


> #3. Upstream gate jobs
>   Ideally, gate jobs should never use ":latest". This is in trunk/dlrn
> terms the equivalent of "/current/" or "/consistent/".
>   They'd use something like ":latest-passed-ci" which would be the
> proper equivalent of "/current-passed-ci/" or "/current-tripleo/".
>
>
There is nothing special about the word latest. Can we give these images
the same tag as the name of the package repo they came from? so
:current-passed-ci :current-tripleo?


>   This brings an interesting challenge around how we currently add new
> images to overcloud_containers.yaml (example [3]).
>   It is expected that, when you add the image, the image is already
> present on the registry because otherwise the container jobs will fail
> since this new image cannot be pulled (example [4]).
>   My understanding currently is that humans may build and push images
> to the registry ahead of time so that this works.
>   We can keep a similar approach if that's what we really want with
> the new private registry, the job that builds container is made
> generic exactly to be able to build just a specific set of image(s) if
> we want.
>   Here's the catch, though: this new container image will have the
> ":latest" tag, it will not have ":latest-passed-ci" because it hasn't
> passed CI yet, it's being added just now.
>   So how do we address this ?
>
>
Here is an idea, the "discover" command mentioned above could filter images
based on their presence in the registry with the required tags, so the
resulting generated overcloud_containers.yaml would have less entries if
there is no image with the requested tag.


>   Note:
>   We've already discussed that some containers need to pick up the
> latest and the greatest from the "/current/" repository, either
> because they are "direct" tripleo packages or if "Depends-On" is used.
>   So far, the solution we seem to be going towards is to pick up the
> containers from ":latest-passed-ci" and then more or less add a 'yum
> update' layer to the images needing an update.
>   This is the option that is in the best interest of time, we'd
> otherwise be spending too much time building containers in jobs that
> are already taking way too long to run.
>

That is a shame, I have no suggestions to avoid this though.


> #4. Test days
>   When doing test days, we know to point testers to
> /current-passed-ci/ as well as tested quickstart images.
>   How can we make it easy for containers ? If the container list from
> tripleo-common is hardcoded to latest, that won't work. If it's
> hardcoded to :latest-passed-ci, it won't work for other use cases.
>   Ideally this would be super easy for end users as well as developers
> so that they can get started easily.
>
> #5 Stable releases, users, operators & co
>   The packaging workflow is not the same for trunk (dlrn out of git
> source on trunk.rdoproject.oirg) and for stable releases (CentOS build
> system out of released tarballs on mirror.centos.org).
>   It's also going to be different for containers.
>   For trunk, we'll be building containers with trunk repositories and
> publishing them to a private registry analogous to
> trunk.rdoproject.org repositories.
>   For stable releases, while still hand-wavy and foggy, we seem to be
> headed in the direction of the CentOS official registry which takes
> Dockerfiles in pseudo dist-git repositories and builds/publishes the
> containers through Jenkins jobs.
>   This is sort of similar to how downstream would work if you replace
> Jenkins by Brew/OSBS.
>
>   So here, we want to use the overcloud_containers.yaml file to
> "compile" Dockerfiles which will be shipped to git repositories and
> then built by another process.
>

I was working on a subpackage of tripleo-common which packaged the
dockerfiles generated from a kolla-build --template-only run, but apevec
pointed out an issue due to kolla being a build-time tool, and the order
promotions happened could result in a version mismatch between the
dockerfiles subpackage and the kolla package in that repo.

If we can't do it in an RPM then we'll need to figure out some other
mechanism to generate static dockerfiles whenever tripleo-common or kolla
rpms change. Maybe a git repo which is populated with commits by an
automated job is the best approach (pseudo dist-git)


>   These containers will be published somewhere that tripleo-common is
> sort of expected to know ahead of time because users, customers and
> developers need to be pulling from that "stable" source and from a
> "stable" tag.
>
> So... what do we do ?
>
> I think the following would address all your issues, does this sound
reasonable?
1. implement "openstack overcloud container image build --list-images"
2. make the canonical overcloud_containers.yaml a template
3. implement "openstack overcloud container image discover"
4. implement a pseudo dist-git populated with generated dockerfiles for
other image building tools

I'd be interested in working on 1, 2, 3


> [1]: https://ci.centos.org/view/rdo/view/promotion-pipeline/
> job/rdo_trunk-promote-master-current-tripleo/
> [2]: http://i.imgur.com/I26njTz.png
> [3]: https://review.openstack.org/#/c/466305/
> [4]: http://logs.openstack.org/51/467651/2/check-tripleo/gate-
> tripleo-ci-centos-7-ovb-containers-oooq-nv/dc61632/
> logs/oooq/undercloud/home/jenkins/overcloud_prep_
> containers.log.txt.gz#_2017-05-24_19_24_13
>
> David Moreau Simard
> Senior Software Engineer | Openstack RDO
>
> dmsimard = [irc, github, twitter]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170530/3bcf117c/attachment.html>