<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 27, 2017 at 7:07 AM, David Moreau Simard <span dir="ltr"><<a href="mailto:dms@redhat.com" target="_blank">dms@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
Today we discussed a challenge around image tags that mostly boils<br>
down to limitations in how overcloud_containers.yaml is constructed<br>
and used.<br>
<br>
TL;DR, we need a smart and easy way to work with the<br>
overcloud_containers.yaml file (especially tags).<br>
<br>
Let's highlight a few use cases that we need to work through:<br>
<br>
#1. Building containers<br>
For building containers, all we really care is the name of the<br>
images we need to build.<br>
Today, we install a trunk repository and then install<br>
tripleo-common-containers from that trunk repository.<br>
We then mostly grep/sed/awk our way from overcloud_containers.yaml<br>
to a clean list of images to build and then build those.<br>
Relatively okay with this but prone to things breaking -- a clean<br>
way to get just the list of images out of there would be nice.<br>
<br></blockquote><div><br></div><div>The command "openstack overcloud container image build" also has some string matching logic, but then invokes kolla-build directly.</div><div><br></div><div>Can I suggest that we add a --list-images option to this command so that it just returns a list of images for other image building tools to consume?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
#2. Testing and promoting containers<br>
This comes right after use case #1 where we build containers in the pipeline.<br>
For those familiar with the CI pipeline to do promotions [1], this<br>
would look a bit like this [2].<br>
<br>
In practice, this works basically the same way as we build, test and<br>
promote tripleo-quickstart images.<br>
We pick a particular trunk repository hash and build containers for<br>
that hash. These are then pushed with both the tags ":latest" and<br>
":<dlrn_hash>".<br>
We're then supposed to test those containers in the pipeline but to<br>
do that, we need to be pulling from :<dlrn_hash>, not :latest...<br>
although they are in theory equivalent at that given time, this might<br>
not always be true.<br>
So the testing job(s) need a way to easily customize/pull from that<br>
particular hash instead of the hardcoded latest we have right now.<br>
<br></blockquote><div><br></div><div>I would like to see another "openstack overcloud container image ..." command which is pointed at an image registry and a canonical overcloud_containers.yaml file, then generates another overcloud_containers.yaml (and heat environment file) which contains the proper latest <dlrn_hash> tags. This tool could work too for stable version-style tags.<br></div><div><br></div><div>How about "openstack overcloud container image discover"?</div><div><br></div><div>This would be easier to implement if the canonical overcloud_containers.yaml file was a template rather than a file with hard-coded namespace and tags.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
#3. Upstream gate jobs<br>
Ideally, gate jobs should never use ":latest". This is in trunk/dlrn<br>
terms the equivalent of "/current/" or "/consistent/".<br>
They'd use something like ":latest-passed-ci" which would be the<br>
proper equivalent of "/current-passed-ci/" or "/current-tripleo/".<br>
<br></blockquote><div><br></div><div>There is nothing special about the word latest. Can we give these images the same tag as the name of the package repo they came from? so :current-passed-ci :current-tripleo?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
This brings an interesting challenge around how we currently add new<br>
images to overcloud_containers.yaml (example [3]).<br>
It is expected that, when you add the image, the image is already<br>
present on the registry because otherwise the container jobs will fail<br>
since this new image cannot be pulled (example [4]).<br>
My understanding currently is that humans may build and push images<br>
to the registry ahead of time so that this works.<br>
We can keep a similar approach if that's what we really want with<br>
the new private registry, the job that builds container is made<br>
generic exactly to be able to build just a specific set of image(s) if<br>
we want.<br>
Here's the catch, though: this new container image will have the<br>
":latest" tag, it will not have ":latest-passed-ci" because it hasn't<br>
passed CI yet, it's being added just now.<br>
So how do we address this ?<br>
<br></blockquote><div><br></div><div>Here is an idea, the "discover" command mentioned above could filter images based on their presence in the registry with the required tags, so the resulting generated overcloud_containers.yaml would have less entries if there is no image with the requested tag.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Note:<br>
We've already discussed that some containers need to pick up the<br>
latest and the greatest from the "/current/" repository, either<br>
because they are "direct" tripleo packages or if "Depends-On" is used.<br>
So far, the solution we seem to be going towards is to pick up the<br>
containers from ":latest-passed-ci" and then more or less add a 'yum<br>
update' layer to the images needing an update.<br>
This is the option that is in the best interest of time, we'd<br>
otherwise be spending too much time building containers in jobs that<br>
are already taking way too long to run.<br></blockquote><div><br></div><div>That is a shame, I have no suggestions to avoid this though.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
#4. Test days<br>
When doing test days, we know to point testers to<br>
/current-passed-ci/ as well as tested quickstart images.<br>
How can we make it easy for containers ? If the container list from<br>
tripleo-common is hardcoded to latest, that won't work. If it's<br>
hardcoded to :latest-passed-ci, it won't work for other use cases.<br>
Ideally this would be super easy for end users as well as developers<br>
so that they can get started easily.<br>
<br>
#5 Stable releases, users, operators & co<br>
The packaging workflow is not the same for trunk (dlrn out of git<br>
source on trunk.rdoproject.oirg) and for stable releases (CentOS build<br>
system out of released tarballs on <a href="http://mirror.centos.org" rel="noreferrer" target="_blank">mirror.centos.org</a>).<br>
It's also going to be different for containers.<br>
For trunk, we'll be building containers with trunk repositories and<br>
publishing them to a private registry analogous to<br>
<a href="http://trunk.rdoproject.org" rel="noreferrer" target="_blank">trunk.rdoproject.org</a> repositories.<br>
For stable releases, while still hand-wavy and foggy, we seem to be<br>
headed in the direction of the CentOS official registry which takes<br>
Dockerfiles in pseudo dist-git repositories and builds/publishes the<br>
containers through Jenkins jobs.<br>
This is sort of similar to how downstream would work if you replace<br>
Jenkins by Brew/OSBS.<br>
<br>
So here, we want to use the overcloud_containers.yaml file to<br>
"compile" Dockerfiles which will be shipped to git repositories and<br>
then built by another process.<br></blockquote><div><br></div><div>I was working on a subpackage of tripleo-common which packaged the dockerfiles generated from a kolla-build --template-only run, but apevec pointed out an issue due to kolla being a build-time tool, and the order promotions happened could result in a version mismatch between the dockerfiles subpackage and the kolla package in that repo.</div><div><br></div><div>If we can't do it in an RPM then we'll need to figure out some other mechanism to generate static dockerfiles whenever tripleo-common or kolla rpms change. Maybe a git repo which is populated with commits by an automated job is the best approach (pseudo dist-git)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
These containers will be published somewhere that tripleo-common is<br>
sort of expected to know ahead of time because users, customers and<br>
developers need to be pulling from that "stable" source and from a<br>
"stable" tag.<br>
<br>
So... what do we do ?<br>
<br></blockquote><div>I think the following would address all your issues, does this sound reasonable?</div><div>1. implement "openstack overcloud container image build --list-images"</div><div>2. make the canonical overcloud_containers.yaml a template</div><div>3. implement "openstack overcloud container image discover"</div><div>4. implement a pseudo dist-git populated with generated dockerfiles for other image building tools </div><div><br></div><div>I'd be interested in working on 1, 2, 3</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
[1]: <a href="https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/" rel="noreferrer" target="_blank">https://ci.centos.org/view/<wbr>rdo/view/promotion-pipeline/<wbr>job/rdo_trunk-promote-master-<wbr>current-tripleo/</a><br>
[2]: <a href="http://i.imgur.com/I26njTz.png" rel="noreferrer" target="_blank">http://i.imgur.com/I26njTz.png</a><br>
[3]: <a href="https://review.openstack.org/#/c/466305/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/466305/</a><br>
[4]: <a href="http://logs.openstack.org/51/467651/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/dc61632/logs/oooq/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-05-24_19_24_13" rel="noreferrer" target="_blank">http://logs.openstack.org/51/<wbr>467651/2/check-tripleo/gate-<wbr>tripleo-ci-centos-7-ovb-<wbr>containers-oooq-nv/dc61632/<wbr>logs/oooq/undercloud/home/<wbr>jenkins/overcloud_prep_<wbr>containers.log.txt.gz#_2017-<wbr>05-24_19_24_13</a><br>
<br>
David Moreau Simard<br>
Senior Software Engineer | Openstack RDO<br>
<br>
dmsimard = [irc, github, twitter]<br>
</blockquote></div><br></div></div>