Open Stack

On Thu, Jan 12, 2017 at 03:02:32PM -0000, Edmund Rhudy (BLOOMBERG/ 120 PARK) wrote:
> Here at Bloomberg, we're evaluating Kolla to replace our in-house OpenStack deployment system, and one of our requirements is that we be able to do our builds without touching the Internet - everything needs to come from locally hosted repositories. A few weeks ago, I pushed up a PR (https://review.openstack.org/#/c/414639/) to start working on the ability to build Kolla containers while disconnected from the Internet. It doesn't provide complete coverage by any means (though that is the goal, to ensure that every container can be built offline for every base OS image), but I wanted to use it as a starter for further discussion, as well as reducing the amount of stuff we're carrying as local changes on top of upstream Kolla.
> 
> That being said, when I pushed the PR up, it failed the Ubuntu checks. I looked into it, and here's what I found:
> 
> 1) There is a bug in Kolla (https://bugs.launchpad.net/kolla/+bug/1633187) that causes it to ignore any custom sources.list provided when building Debian/Ubuntu containers. You can supply one, and it will be copied into the build context, but because of http://git.openstack.org/cgit/openstack/kolla/tree/docker/base/Dockerfile.j2#n215, only the sources.list files that come with Kolla would be used anyway. Necessarily, because using local mirrors requires providing a custom sources.list, I fixed this bug (https://bugs.launchpad.net/kolla/+bug/1633187).
> 
> 2) The Ubuntu gate checks provide a custom sources.list which redirects the container away from Canonical's mirrors and onto OSIC-hosted mirrors. The OSIC mirror, for whatever reason, is unsigned. In current master Kolla, this sources.list just isn't used, so checks that rebuild the base image will always use archive.ubuntu.com, because that's the mirror that's specified in docker/base/sources.list.ubuntu. Take for example the output of another PR https://review.openstack.org/#/c/411154/ - if you examine http://logs.openstack.org/54/411154/12/check/gate-kolla-dsvm-build-ubuntu-binary-ubuntu-xenial-nv/26627d8/console.html.gz (from the very top), you can see that it's downloading packages from archive.ubuntu.com as part of the base container build, even though http://logs.openstack.org/54/411154/12/check/gate-kolla-dsvm-build-ubuntu-binary-ubuntu-xenial-nv/26627d8/logs/kolla_configs/kolla/sources.list.txt.gz is supplied as sources.list.
> 
This are not OSIC mirrors, but in fact openstack-infra mirrors (stored in AFS).
You are also correct, the packages are not signed, as this is not an official
mirror. It is something in openstack-infra we has talked about fixing, but
haven't done the work for it yet.

> 3) When I fixed the bug described in #1, it meant the unsigned OSIC mirror specified in sources.list suddenly started getting used, and the base container build now fails because the container build process does not allow unauthenticated packages to be installed.
> 
> How can this be fixed? There are a few options:
> 
> 1) Remove the sources.list from the current gate configurations - the way things are currently set up, the Ubuntu gates actually _depend_ on the presence of a bug in Kolla to function if they ever need to build the base Kolla image. This is not good.
> 
> 2) I don't know why the OSIC Ubuntu mirror is unsigned. I feel like it should be a straight clone of Canonical's repos so that the baked-in signing key for the Ubuntu base image will just work, but presumably it's this way for a reason?
> 
The actually reason, we use reprepro[1] for the mirroring. Because we don't want
to release broken mirrors into our test workers, we do some things during the
mirror process.  Specifically, we don't delete packages on the first mirror
attempt.  We log which packages need to be deleted, then on the next loop (2
hours later) we purge those packages from the system.  As a results, we cannot
preserve the gpg keys from upstream.

[1] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/reprepro/reprepro-mirror-update.sh

> 3) Specify a custom apt preferences in the gate to allow installing unauthenticated packages in the containers (ugly).
> 
Yes, this is what we do on all images today[2]. It is not ugly, just want is
needed to make our mirrors work in the gate.

[2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n194

> Would somebody with knowledge of the Kolla testing infrastructure be so kind as to comment? I brought this up in IRC a few times but could not get much attention on it.
I would recommend joining #openstack-infra to get more information.

I cannot speak for kolla, but adding the changes I suggested fixes your issues.
However, I wouldn't use the ubuntu containers in production, until you rebuilt
them yourself using your local infrastruture.

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev