Open Stack

Hi all,

I accidentally spent last day looking into some timings of kolla-build.
I'm sharing my results below - perhaps it will help somebody:

I've measured time for 4 consecutive kolla-builds (on a decent laptop, 
with modified default profile including 
"kolla-toolbox,glance,horizon,keystone,mariadb,memchached,neutron,nova,rabbitmq,chronos,marathon,mesos-master,mesos-slave,zookeeper", 
-b ubuntu -t binary):

real    25m20.867s  (fresh environment, all Docker images wiped out)
real    22m54.298s  (2nd run, all Docker images from previous run in place)
real    25m52.602s  (3nd run, all Docker images from previous run in 
place, local apt-cacher-ng enabled for the 1st time and stored 300MB of 
DEBs)
real    18m28.739s  (4th run, all DEBs were collected from apt-cacher = 
local disk)

To compare, kolla-build run on a server (with --threads 6 option) takes 
about 15 minutes.
I've tweaked mirrors for Ubuntu but I haven't observed any substantial 
difference.
BTW it may not be a good idea to start >1 concurrent builds on 1 Jenkins 
slave (due to high load).

CONCLUSIONS:
* kolla-build script (or tools it's using, e.g. Docker registry) doesn't 
seem to be making good usage of Docker cache mechanisms (or they are 
broken themselves) and I think is should be investigated/fixed
* having DEBs close or in cache seems like a promising approach that 
never hurts: IMO we should deploy it and start saving time & bandwidth
* stable bandwidth of roughly 30 Mbit/s is needed to match building 
Kolla locally (on a laptop or lab, comparing to CI). With bandwidth of 
like 200Mbit/s developer could get images in his lab in like 17 minutes 
from the moment he commited a patch. This can be further optimized with 
rebuilding just changed images explicitly (argument to kolla-build) but 
I investigated this & IMO it's not trivial b/c of various corner cases 
if we want to provide a complete set of images for kolla-mesos-deploy 
(under 1 Docker tag) - I can elaborate if you want.
* if we want to use CI for image building & deployment tests we need 
strong infra with local Docker registry in it, all in 1 network (I mean 
CI slaves, registry & perhaps even lab for some manual tests).

Again guys, these are just quick&dirty results but I kind of sense "tip 
of an iceberg" problem here.

-marek


On 23.02.2016 17:58, Steven Dake (stdake) wrote:
> Ricardo,
>
> Apologies in lag, I remember reading your email but not answering
> personally.  I thought Michal's response was appropriate but there is a
> bit more to it.
>
> We definitely want to have images for the following events:
> * Any tag such as a milestone, or release candidate, or release
> * Nightly builds for folks that want to run bleeding edge without the pain
> of building images
>
> The nightly builds can easily be culled after 30 days.
>
> We have implementations for centos, ubuntu, and oracle linux.  We have
> source and binary versions of both these distros, so essentially we have 6
> sets of registry images.
>
> There are 115 containers we would build, push, and tag.  I think Michal's
> estimate is for a limited subset of the containers.  Ideally we would want
> to build and push all the containers in our repository, which I believe is
> 10-15GB per distro+build type combination.
>
> At the high end, we are looking at
> * (for permanently tagged releases) 15gb*6 sets of images * (3 milestone
> tags + 1 release tag + 1-3 RC's per cycle = 7) * 2 release cycles per year
> = 1 terabyte of images per cycle growing approximately ~ 1.3 terabytes per
> year
> * (for the nightly builds), 15bb*6 sets of images * 30 = 2.7 terabytes in
> continuous use.
>
> So that is 2.2 TB baseline + 1TB per year growth.  As we add containers
> over time, the yearly growth may increase, but I doubt would ever be more
> then 2 terabytes in the next 4 release cycles of OpenStack.
>
> There are several registry storage backends available including swift, so
> if infra has swift available, that would be a viable option.
>
> I'll get an exact number for our containers as they build today and
> respond to this thread since it affects the estimate.
>
> It is not critical the storage be SSD, since tag and nightly build
> operations could happen with long run-time on the post jobs I believe
> without harming infra resources (although they may go past the 90 minute
> limit infra likes to stick to without SSD).  I don't have any non-SSD to
> test the build with, so I have no idea what performance impact the SSDs
> have.  I know when I went from regular SSD to PCI-E NVM SSD (Intel 750) my
> build times dropped from 50 minutes to about 15 minutes.
>
> Note we don't have gate jobs at present for oracle linux nor Ubuntu
> binary.  Only CentOS binary, CentOS source, and Ubuntu source are in our
> gates at present.  For the present term, storage would be 630GB per year.
>
> I'll get back to you with exact numbers on storage used by the registry in
> a full build scenario today.
>
> Regards
> -steve
>
>
> On 2/21/16, 9:26 AM, "Michał Jastrzębski" <inc007 at gmail.com> wrote:
>
>> I'd say 5Gigs should be enough for all the images per distro (maybe
>> less if we have to squeeze). Since we have have 2 strongly supported
>> distro 10Gigs. If we would like to add all distros we support, that's
>> 20-25 (I think). That also depends how many older versions we want to
>> keep (current+stable would be absolute minimum, we might increase it
>> to milestones). We have lot's of options to tweak so no one will get
>> hurt, and if we have dedicated machine for us (which we should because
>> apart from disk space, registry can actually eat up lots of IOPS, can
>> be VM tho with disk that can handle that), I think any dedicated,
>> industry standard, disk should be enough (but SSD would be great).
>>
>> Cheers,
>> Michal
>>
>> On 20 February 2016 at 16:14, Ricardo Carrillo Cruz
>> <ricardo.carrillo.cruz at gmail.com> wrote:
>>> Hi Steve
>>>
>>> When you say the registry would require a machine with plenty of disk
>>> space,
>>> do you have an estimate of storage needed?
>>>
>>> Regards
>>>
>>> 2016-02-20 14:21 GMT+01:00 Steven Dake (stdake) <stdake at cisco.com>:
>>>> Infra folks,
>>>>
>>>> I'd like to see a full CI/CD pipeline of Kolla to an OpenStack
>>>> infrastructure hosted registry.
>>>>
>>>> With docker registry 2.2 and earlier a Docker push of Kolla containers
>>>> took 5-10 hours.  This is because of design problems in Docker which
>>>> made a
>>>> push each layer of each Docker image repeatedly.  This has been
>>>> rectified in
>>>> docker-regitery 2.3 (the latest hub tagged docker registry).  The 5-10
>>>> hour
>>>> upload times are now down to about 15 minutes.  Now it takes
>>>> approximately
>>>> 15 minutes to push all 115 kolla containers on a gigabit network.
>>>>
>>>> Kolla in general wants to publish to a docker registry at least per
>>>> tag,
>>>> and possibly per commit (or alternatively daily).  We already build
>>>> Kolla
>>>> images in the gate, and although sometimes our jobs time out on CentOS
>>>> the
>>>> build on Ubuntu is about 12 minutes.  The reason our jobs time out on
>>>> CentOS
>>>> is because we lack local to the infrastructure mirrors as is available
>>>> on
>>>> Ubuntu from a recent patch I believe that Monty offered.
>>>>
>>>> We have one of two options going forward
>>>>
>>>> We could publish to the docker hub registry
>>>> We could publish to docker-registry.openstack.org
>>>>
>>>> Having a docker-registry.openstack.org would be my preference, but
>>>> requires a machine with plenty of disk space and a copy of docker
>>>> 1.10.1 or
>>>> later running on it.  The docker-registry 2.3 and later runs as a
>>>> container
>>>> inside Docker.  The machine could be Ubuntu or CentOS - we have gate
>>>> scripts
>>>> for both that do the machine setup which the infrastructure team could
>>>> begin
>>>> with[1][2]  I don't care which distro is used for docker registry - it
>>>> reallly shouldn't matter as it will be super lightweight and really
>>>> only
>>>> need a /var/lib/docker that is fast and large.  Kolla dev's can help
>>>> get the
>>>> docker registry setup and provide guidance to the infrastructure team
>>>> on how
>>>> to setup Docker, but I'm unclear of OpenStack has resources to make
>>>> this
>>>> particular request happen.
>>>>
>>>> NB the machine need not be baremetal - it  really doesn't matter.  It
>>>> does
>>>> need fast bi-directional networking and fast disk IO to meet the gate
>>>> timeout requirements and Operator requirements that a pull is speedy.
>>>> The
>>>> other change needed is a CentOS mirror internal to the infrastructure,
>>>> so
>>>> our CentOS jobs don't time out and we can push per cmmit (or we could
>>>> add a
>>>> nightly job).
>>>>
>>>> This is something new OpenStack hasn't done before, so feedback from
>>>> the
>>>> infrastructure team welcome if that team is willing to maintain a
>>>> docker-registry.openstack.org.  The other challenge here will be
>>>> authentication - we setup our gate Docker without TLS because we throw
>>>> away
>>>> the VMs but infra will want to setup TLS with the docker registry.
>>>> Folks
>>>> wanting to use the docker reigstry service from OpenStack will need to
>>>> be
>>>> able to put TLS credentials in the gating in some way.  I'm not sure
>>>> we want
>>>> to just check these credentials into our repository - which means they
>>>> need
>>>> to somehow be injected into our VMs to protect the security of the
>>>> Docker
>>>> images.
>>>>
>>>> If infra decides they don't want to take on a
>>>> docker-registry.openstack.org, guidance on how to get our credentials
>>>> securely into our built VM would be helpful.
>>>>
>>>> One final note - Docker can be setup to use Swift as a storage
>>>> backend, or
>>>> alternatively can use straight up disk space on the node.  It can also
>>>> publish to an AWS storage backend and has many other storage backend
>>>> modes.
>>>>
>>>> Regards
>>>> -steve
>>>>
>>>>
>>>> [1]
>>>> https://github.com/openstack/kolla/blob/master/tools/setup_RedHat.sh
>>>> [2]
>>>> https://github.com/openstack/kolla/blob/master/tools/setup_Debian.sh
>>>>
>>>>
>>>> ________________________________________________________________________
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> _
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Marek Zawadzki
Mirantis Kolla Team