Open Stack

Ricardo,

Apologies in lag, I remember reading your email but not answering
personally.  I thought Michal's response was appropriate but there is a
bit more to it.

We definitely want to have images for the following events:
* Any tag such as a milestone, or release candidate, or release
* Nightly builds for folks that want to run bleeding edge without the pain
of building images

The nightly builds can easily be culled after 30 days.

We have implementations for centos, ubuntu, and oracle linux.  We have
source and binary versions of both these distros, so essentially we have 6
sets of registry images.

There are 115 containers we would build, push, and tag.  I think Michal's
estimate is for a limited subset of the containers.  Ideally we would want
to build and push all the containers in our repository, which I believe is
10-15GB per distro+build type combination.

At the high end, we are looking at
* (for permanently tagged releases) 15gb*6 sets of images * (3 milestone
tags + 1 release tag + 1-3 RC's per cycle = 7) * 2 release cycles per year
= 1 terabyte of images per cycle growing approximately ~ 1.3 terabytes per
year
* (for the nightly builds), 15bb*6 sets of images * 30 = 2.7 terabytes in
continuous use.

So that is 2.2 TB baseline + 1TB per year growth.  As we add containers
over time, the yearly growth may increase, but I doubt would ever be more
then 2 terabytes in the next 4 release cycles of OpenStack.

There are several registry storage backends available including swift, so
if infra has swift available, that would be a viable option.

I'll get an exact number for our containers as they build today and
respond to this thread since it affects the estimate.

It is not critical the storage be SSD, since tag and nightly build
operations could happen with long run-time on the post jobs I believe
without harming infra resources (although they may go past the 90 minute
limit infra likes to stick to without SSD).  I don't have any non-SSD to
test the build with, so I have no idea what performance impact the SSDs
have.  I know when I went from regular SSD to PCI-E NVM SSD (Intel 750) my
build times dropped from 50 minutes to about 15 minutes.

Note we don't have gate jobs at present for oracle linux nor Ubuntu
binary.  Only CentOS binary, CentOS source, and Ubuntu source are in our
gates at present.  For the present term, storage would be 630GB per year.

I'll get back to you with exact numbers on storage used by the registry in
a full build scenario today.

Regards
-steve


On 2/21/16, 9:26 AM, "Michał Jastrzębski" <inc007 at gmail.com> wrote:

>I'd say 5Gigs should be enough for all the images per distro (maybe
>less if we have to squeeze). Since we have have 2 strongly supported
>distro 10Gigs. If we would like to add all distros we support, that's
>20-25 (I think). That also depends how many older versions we want to
>keep (current+stable would be absolute minimum, we might increase it
>to milestones). We have lot's of options to tweak so no one will get
>hurt, and if we have dedicated machine for us (which we should because
>apart from disk space, registry can actually eat up lots of IOPS, can
>be VM tho with disk that can handle that), I think any dedicated,
>industry standard, disk should be enough (but SSD would be great).
>
>Cheers,
>Michal
>
>On 20 February 2016 at 16:14, Ricardo Carrillo Cruz
><ricardo.carrillo.cruz at gmail.com> wrote:
>> Hi Steve
>>
>> When you say the registry would require a machine with plenty of disk
>>space,
>> do you have an estimate of storage needed?
>>
>> Regards
>>
>> 2016-02-20 14:21 GMT+01:00 Steven Dake (stdake) <stdake at cisco.com>:
>>>
>>> Infra folks,
>>>
>>> I'd like to see a full CI/CD pipeline of Kolla to an OpenStack
>>> infrastructure hosted registry.
>>>
>>> With docker registry 2.2 and earlier a Docker push of Kolla containers
>>> took 5-10 hours.  This is because of design problems in Docker which
>>>made a
>>> push each layer of each Docker image repeatedly.  This has been
>>>rectified in
>>> docker-regitery 2.3 (the latest hub tagged docker registry).  The 5-10
>>>hour
>>> upload times are now down to about 15 minutes.  Now it takes
>>>approximately
>>> 15 minutes to push all 115 kolla containers on a gigabit network.
>>>
>>> Kolla in general wants to publish to a docker registry at least per
>>>tag,
>>> and possibly per commit (or alternatively daily).  We already build
>>>Kolla
>>> images in the gate, and although sometimes our jobs time out on CentOS
>>>the
>>> build on Ubuntu is about 12 minutes.  The reason our jobs time out on
>>>CentOS
>>> is because we lack local to the infrastructure mirrors as is available
>>>on
>>> Ubuntu from a recent patch I believe that Monty offered.
>>>
>>> We have one of two options going forward
>>>
>>> We could publish to the docker hub registry
>>> We could publish to docker-registry.openstack.org
>>>
>>> Having a docker-registry.openstack.org would be my preference, but
>>> requires a machine with plenty of disk space and a copy of docker
>>>1.10.1 or
>>> later running on it.  The docker-registry 2.3 and later runs as a
>>>container
>>> inside Docker.  The machine could be Ubuntu or CentOS - we have gate
>>>scripts
>>> for both that do the machine setup which the infrastructure team could
>>>begin
>>> with[1][2]  I don't care which distro is used for docker registry - it
>>> reallly shouldn't matter as it will be super lightweight and really
>>>only
>>> need a /var/lib/docker that is fast and large.  Kolla dev's can help
>>>get the
>>> docker registry setup and provide guidance to the infrastructure team
>>>on how
>>> to setup Docker, but I'm unclear of OpenStack has resources to make
>>>this
>>> particular request happen.
>>>
>>> NB the machine need not be baremetal - it  really doesn't matter.  It
>>>does
>>> need fast bi-directional networking and fast disk IO to meet the gate
>>> timeout requirements and Operator requirements that a pull is speedy.
>>>The
>>> other change needed is a CentOS mirror internal to the infrastructure,
>>>so
>>> our CentOS jobs don't time out and we can push per cmmit (or we could
>>>add a
>>> nightly job).
>>>
>>> This is something new OpenStack hasn't done before, so feedback from
>>>the
>>> infrastructure team welcome if that team is willing to maintain a
>>> docker-registry.openstack.org.  The other challenge here will be
>>> authentication - we setup our gate Docker without TLS because we throw
>>>away
>>> the VMs but infra will want to setup TLS with the docker registry.
>>>Folks
>>> wanting to use the docker reigstry service from OpenStack will need to
>>>be
>>> able to put TLS credentials in the gating in some way.  I'm not sure
>>>we want
>>> to just check these credentials into our repository - which means they
>>>need
>>> to somehow be injected into our VMs to protect the security of the
>>>Docker
>>> images.
>>>
>>> If infra decides they don't want to take on a
>>> docker-registry.openstack.org, guidance on how to get our credentials
>>> securely into our built VM would be helpful.
>>>
>>> One final note - Docker can be setup to use Swift as a storage
>>>backend, or
>>> alternatively can use straight up disk space on the node.  It can also
>>> publish to an AWS storage backend and has many other storage backend
>>>modes.
>>>
>>> Regards
>>> -steve
>>>
>>>
>>> [1] 
>>>https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh
>>> [2] 
>>>https://github.com/openstack/kolla/blob/master/tools/setup_Debian.sh
>>>
>>> 
>>>________________________________________________________________________
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> 
>>_________________________________________________________________________
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev