Open Stack

Ricardo,

I measured the disk space used by the docker registry.

For ubuntu images, it is 1.5gb.
For centos images, it is 2.5gb.
For oraclelinux images, it is 2.5gb. (derivative of centos)
The delta between centos and Ubuntu has to do with the usage of apt vs
yum, where apt only installs a minimum set of requirement files.

For all distro and source combinations it is 1.5+1.5+2.5+2.5+2.5+2.5=13
Gbs for all containers (about 500 or so containers)

13gb * (3 release tags + 1 release tag + 3 Rcs per cycle) = 91gb per cycle
13gb * 30 days = 390gb

So the storage requirements are not as bad as originally thought.

Note it takes me 6 minutes to build on my PCIE SSD with gigabit to the
Internet via Cox communications.  It takes 2 minutes to push to a local
registry at 10 gigabit with PCIE SSDs.

Our gate job with mirrors of the APT repos takes between 25 and 30 minutes
to build.  A post job to build and push the commit on tag or nightly is
easy for us to tackle on the kolla end.

Where we are stumped is should we push to registry.openstack.org which I
would prefer, or push to the docker registry.

Another issue is getting the authentication credentials in the gate
securely - somehow it needs to be injected into our post jobs without us
having to check our credentials into the repository.

Regards
-steve



On 2/23/16, 9:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:

>Ricardo,
>
>Apologies in lag, I remember reading your email but not answering
>personally.  I thought Michal's response was appropriate but there is a
>bit more to it.
>
>We definitely want to have images for the following events:
>* Any tag such as a milestone, or release candidate, or release
>* Nightly builds for folks that want to run bleeding edge without the pain
>of building images
>
>The nightly builds can easily be culled after 30 days.
>
>We have implementations for centos, ubuntu, and oracle linux.  We have
>source and binary versions of both these distros, so essentially we have 6
>sets of registry images.
>
>There are 115 containers we would build, push, and tag.  I think Michal's
>estimate is for a limited subset of the containers.  Ideally we would want
>to build and push all the containers in our repository, which I believe is
>10-15GB per distro+build type combination.
>
>At the high end, we are looking at
>* (for permanently tagged releases) 15gb*6 sets of images * (3 milestone
>tags + 1 release tag + 1-3 RC's per cycle = 7) * 2 release cycles per year
>= 1 terabyte of images per cycle growing approximately ~ 1.3 terabytes per
>year
>* (for the nightly builds), 15bb*6 sets of images * 30 = 2.7 terabytes in
>continuous use.
>
>So that is 2.2 TB baseline + 1TB per year growth.  As we add containers
>over time, the yearly growth may increase, but I doubt would ever be more
>then 2 terabytes in the next 4 release cycles of OpenStack.
>
>There are several registry storage backends available including swift, so
>if infra has swift available, that would be a viable option.
>
>I'll get an exact number for our containers as they build today and
>respond to this thread since it affects the estimate.
>
>It is not critical the storage be SSD, since tag and nightly build
>operations could happen with long run-time on the post jobs I believe
>without harming infra resources (although they may go past the 90 minute
>limit infra likes to stick to without SSD).  I don't have any non-SSD to
>test the build with, so I have no idea what performance impact the SSDs
>have.  I know when I went from regular SSD to PCI-E NVM SSD (Intel 750) my
>build times dropped from 50 minutes to about 15 minutes.
>
>Note we don't have gate jobs at present for oracle linux nor Ubuntu
>binary.  Only CentOS binary, CentOS source, and Ubuntu source are in our
>gates at present.  For the present term, storage would be 630GB per year.
>
>I'll get back to you with exact numbers on storage used by the registry in
>a full build scenario today.
>
>Regards
>-steve
>
>
>On 2/21/16, 9:26 AM, "Michał Jastrzębski" <inc007 at gmail.com> wrote:
>
>>I'd say 5Gigs should be enough for all the images per distro (maybe
>>less if we have to squeeze). Since we have have 2 strongly supported
>>distro 10Gigs. If we would like to add all distros we support, that's
>>20-25 (I think). That also depends how many older versions we want to
>>keep (current+stable would be absolute minimum, we might increase it
>>to milestones). We have lot's of options to tweak so no one will get
>>hurt, and if we have dedicated machine for us (which we should because
>>apart from disk space, registry can actually eat up lots of IOPS, can
>>be VM tho with disk that can handle that), I think any dedicated,
>>industry standard, disk should be enough (but SSD would be great).
>>
>>Cheers,
>>Michal
>>
>>On 20 February 2016 at 16:14, Ricardo Carrillo Cruz
>><ricardo.carrillo.cruz at gmail.com> wrote:
>>> Hi Steve
>>>
>>> When you say the registry would require a machine with plenty of disk
>>>space,
>>> do you have an estimate of storage needed?
>>>
>>> Regards
>>>
>>> 2016-02-20 14:21 GMT+01:00 Steven Dake (stdake) <stdake at cisco.com>:
>>>>
>>>> Infra folks,
>>>>
>>>> I'd like to see a full CI/CD pipeline of Kolla to an OpenStack
>>>> infrastructure hosted registry.
>>>>
>>>> With docker registry 2.2 and earlier a Docker push of Kolla containers
>>>> took 5-10 hours.  This is because of design problems in Docker which
>>>>made a
>>>> push each layer of each Docker image repeatedly.  This has been
>>>>rectified in
>>>> docker-regitery 2.3 (the latest hub tagged docker registry).  The 5-10
>>>>hour
>>>> upload times are now down to about 15 minutes.  Now it takes
>>>>approximately
>>>> 15 minutes to push all 115 kolla containers on a gigabit network.
>>>>
>>>> Kolla in general wants to publish to a docker registry at least per
>>>>tag,
>>>> and possibly per commit (or alternatively daily).  We already build
>>>>Kolla
>>>> images in the gate, and although sometimes our jobs time out on CentOS
>>>>the
>>>> build on Ubuntu is about 12 minutes.  The reason our jobs time out on
>>>>CentOS
>>>> is because we lack local to the infrastructure mirrors as is available
>>>>on
>>>> Ubuntu from a recent patch I believe that Monty offered.
>>>>
>>>> We have one of two options going forward
>>>>
>>>> We could publish to the docker hub registry
>>>> We could publish to docker-registry.openstack.org
>>>>
>>>> Having a docker-registry.openstack.org would be my preference, but
>>>> requires a machine with plenty of disk space and a copy of docker
>>>>1.10.1 or
>>>> later running on it.  The docker-registry 2.3 and later runs as a
>>>>container
>>>> inside Docker.  The machine could be Ubuntu or CentOS - we have gate
>>>>scripts
>>>> for both that do the machine setup which the infrastructure team could
>>>>begin
>>>> with[1][2]  I don't care which distro is used for docker registry - it
>>>> reallly shouldn't matter as it will be super lightweight and really
>>>>only
>>>> need a /var/lib/docker that is fast and large.  Kolla dev's can help
>>>>get the
>>>> docker registry setup and provide guidance to the infrastructure team
>>>>on how
>>>> to setup Docker, but I'm unclear of OpenStack has resources to make
>>>>this
>>>> particular request happen.
>>>>
>>>> NB the machine need not be baremetal - it  really doesn't matter.  It
>>>>does
>>>> need fast bi-directional networking and fast disk IO to meet the gate
>>>> timeout requirements and Operator requirements that a pull is speedy.
>>>>The
>>>> other change needed is a CentOS mirror internal to the infrastructure,
>>>>so
>>>> our CentOS jobs don't time out and we can push per cmmit (or we could
>>>>add a
>>>> nightly job).
>>>>
>>>> This is something new OpenStack hasn't done before, so feedback from
>>>>the
>>>> infrastructure team welcome if that team is willing to maintain a
>>>> docker-registry.openstack.org.  The other challenge here will be
>>>> authentication - we setup our gate Docker without TLS because we throw
>>>>away
>>>> the VMs but infra will want to setup TLS with the docker registry.
>>>>Folks
>>>> wanting to use the docker reigstry service from OpenStack will need to
>>>>be
>>>> able to put TLS credentials in the gating in some way.  I'm not sure
>>>>we want
>>>> to just check these credentials into our repository - which means they
>>>>need
>>>> to somehow be injected into our VMs to protect the security of the
>>>>Docker
>>>> images.
>>>>
>>>> If infra decides they don't want to take on a
>>>> docker-registry.openstack.org, guidance on how to get our credentials
>>>> securely into our built VM would be helpful.
>>>>
>>>> One final note - Docker can be setup to use Swift as a storage
>>>>backend, or
>>>> alternatively can use straight up disk space on the node.  It can also
>>>> publish to an AWS storage backend and has many other storage backend
>>>>modes.
>>>>
>>>> Regards
>>>> -steve
>>>>
>>>>
>>>> [1] 
>>>>https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh
>>>> [2] 
>>>>https://github.com/openstack/kolla/blob/master/tools/setup_Debian.sh
>>>>
>>>> 
>>>>_______________________________________________________________________
>>>>_
>>>>__
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: 
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>> 
>>>________________________________________________________________________
>>>_
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>_________________________________________________________________________
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev