Open Stack

On 18 May 2017 at 08:03, Paul Belanger <pabelanger at redhat.com> wrote:
> On Tue, May 16, 2017 at 02:11:18PM +0000, Sam Yaple wrote:
>> I would like to bring up a subject that hasn't really been discussed in
>> this thread yet, forgive me if I missed an email mentioning this.
>>
>> What I personally would like to see is a publishing infrastructure to allow
>> pushing built images to an internal infra mirror/repo/registry for
>> consumption of internal infra jobs (deployment tools like kolla-ansible and
>> openstack-ansible). The images built from infra mirrors with security
>> turned off are perfect for testing internally to infra.
>>
> Zuulv3 should have a little with this, it will allow for DAG graph for jobs,
> which means the top level job could be an image build then all jobs below can
> now consume said image.  The steps we are still working on is artifact handling
> but long term, it should be possible for the testing jobs to setup the dynamic
> infrastructure needed themselves.
>
>> If you build images properly in infra, then you will have an image that is
>> not security checked (no gpg verification of packages) and completely
>> unverifiable. These are absolutely not images we want to push to
>> DockerHub/quay for obvious reasons. Security and verification being chief
>> among them. They are absolutely not images that should ever be run in
>> production and are only suited for testing. These are the only types of
>> images that can come out of infra.
>>
> We disable gpg for Ubuntu packaging for a specific reason, most this is because
> our APT repos are not official mirrors of upstream. We regenerate indexes every
> 2 hours as not to break long running jobs.  We have talked in the past of fixing
> this, but it requires openstack-infra to move to a new mirroring tool for APT.

So idea to solve this particular problem goes like this:

Publish job is not a change-driven, it'll be periodical (24h?) during
low time. Then in this job we can turn off using infra mirrors and
just use upstream signed.

That being said, all the technical issues we saw so far (unless I'm
missing something) are solvable and we (kolla community) would love to
do all the heavy lifting to solve it. We need to wait for TC to
resolve non-technical issues before we can proceed tho.

>> Thanks,
>> SamYaple
>>
>> On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski <inc007 at gmail.com>
>> wrote:
>>
>> > On 16 May 2017 at 06:22, Doug Hellmann <doug at doughellmann.com> wrote:
>> > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> > >> Flavio Percoco wrote:
>> > >> > From a release perspective, as Doug mentioned, we've avoided
>> > releasing projects
>> > >> > in any kind of built form. This was also one of the concerns I raised
>> > when
>> > >> > working on the proposal to support other programming languages. The
>> > problem of
>> > >> > releasing built images goes beyond the infrastructure requirements.
>> > It's the
>> > >> > message and the guarantees implied with the built product itself that
>> > are the
>> > >> > concern here. And I tend to agree with Doug that this might be a
>> > problem for us
>> > >> > as a community. Unfortunately, putting your name, Michal, as contact
>> > point is
>> > >> > not enough. Kolla is not the only project producing container images
>> > and we need
>> > >> > to be consistent in the way we release these images.
>> > >> >
>> > >> > Nothing prevents people for building their own images and uploading
>> > them to
>> > >> > dockerhub. Having this as part of the OpenStack's pipeline is a
>> > problem.
>> > >>
>> > >> I totally subscribe to the concerns around publishing binaries (under
>> > >> any form), and the expectations in terms of security maintenance that it
>> > >> would set on the publisher. At the same time, we need to have images
>> > >> available, for convenience and testing. So what is the best way to
>> > >> achieve that without setting strong security maintenance expectations
>> > >> for the OpenStack community ? We have several options:
>> > >>
>> > >> 1/ Have third-parties publish images
>> > >> It is the current situation. The issue is that the Kolla team (and
>> > >> likely others) would rather automate the process and use OpenStack
>> > >> infrastructure for it.
>> > >>
>> > >> 2/ Have third-parties publish images, but through OpenStack infra
>> > >> This would allow to automate the process, but it would be a bit weird to
>> > >> use common infra resources to publish in a private repo.
>> > >>
>> > >> 3/ Publish transient (per-commit or daily) images
>> > >> A "daily build" (especially if you replace it every day) would set
>> > >> relatively-limited expectations in terms of maintenance. It would end up
>> > >> picking up security updates in upstream layers, even if not immediately.
>> > >>
>> > >> 4/ Publish images and own them
>> > >> Staff release / VMT / stable team in a way that lets us properly own
>> > >> those images and publish them officially.
>> > >>
>> > >> Personally I think (4) is not realistic. I think we could make (3) work,
>> > >> and I prefer it to (2). If all else fails, we should keep (1).
>> > >>
>> > >
>> > > At the forum we talked about putting test images on a "private"
>> > > repository hosted on openstack.org somewhere. I think that's option
>> > > 3 from your list?
>> > >
>> > > Paul may be able to shed more light on the details of the technology
>> > > (maybe it's just an Apache-served repo, rather than a full blown
>> > > instance of Docker's service, for example).
>> >
>> > Issue with that is
>> >
>> > 1. Apache served is harder to use because we want to follow docker API
>> > and we'd have to reimplement it
>> > 2. Running registry is single command
>> > 3. If we host in in infra, in case someone actually uses it (there
>> > will be people like that), that will eat up lot of network traffic
>> > potentially
>> > 4. With local caching of images (working already) in nodepools we
>> > loose complexity of mirroring registries across nodepools
>> >
>> > So bottom line, having dockerhub/quay.io is simply easier.
>> >
>> > > Doug
>> > >
>> > > ____________________________________________________________
>> > ______________
>> > > OpenStack Development Mailing List (not for usage questions)
>> > > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>> > unsubscribe
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev