[Openstack-operators] Openstack team size vs's deployment size

Jonathan D. Proulx jon at csail.mit.edu
Tue Sep 13 14:34:56 UTC 2016

On Mon, Sep 12, 2016 at 03:26:02PM -0700, Clint Byrum wrote:
:Excerpts from Mathieu Gagné's message of 2016-09-12 17:55:34 -0400:

:> I also think that small team with small deployments has little
:> incentive to invest in *heavy* automation (to help themselves) and/or
:> tools to delegate its operation to a 3rd party or team. Your
:> deployment isn't "big enough" so you feel it's not worth the
:> investment because "I can manage those just fine" or "There isn't
:> enough compute nodes to automate their installation", etc.
:> Once you hit a certain size, you need to have those in place. Without
:> those tools, you will feel overwhelm by the task, feel like you are
:> the only ones capable of managing/operating the infra and can't
:> delegate to anyone.
:^^ This, so very this.

Yes! Hopefully I'm preaching to the choir here, but I do still hear
about people running with little to no deployment automation 

We basicly got this for 'free' when depolying OpenStack since we'd had
automated network installation and config management for more than a
decade before OpenStack existed (though not the same stack the whole
time) so most of the scafolding was already done. In the immediate
context of OpenStack staffing it's very hard to determine howmuch of
ongoing deployment and config management work is OpenStack costs since
most of it covers a much wider field.

My first OpenStack deploy was Essex on Ununtu12.04 on 60 nodes. Went
from 'Let's try and install OpenStack' to early adopters using it to
complete a tight deadline in less than a week with just one person
working on it.  

This was 99% because we already had Puppet setup for config managemnt
and already had a customized auto deployment for Ubuntu across many
different types of systems (different functions and different
administrative subdomains) so adding another couple types was a well
worn path.

This happened to be the most mature (if I can use that word for Essex)
and tested pairing at the time so it was relatively easy to leverage
other people's work and plug it into our preexisting infrastructure
and work flows. If I had to manually install all those nodes and scp
around lovingly hand crafted configs I don't think it would ever have
worked. Today in OpenStack land most distros and managemnt tools have
pretty good coverage.

You are never too small for automation.  

We have some 'one off' services that are easier to manually poke than
go through updating config managemnt. These tend to drift away from
reproducibility as they age and become terrible burdens to upgrade
since making a replica to test on becomes guess work or sloppy
'restore form back up and make sure you fix all reference to IP and
DNS name where you need ot but not where you shouldn't"

I'm as guilty as anyone for this, but if you don't automate even if
it's unique system you are going to have a bad time.

Conversely pre-OpenStack I built up a 5 server cluster that was the
core internal services for a local start up (with some Xen based
virtualizaton on top so it was more than 5 servers from the outside
but not many more). The first thing I built was the install/management
server and used that to build the 'real services' they asked for.  Now
if I were a good consultant perhaps I'd have worked less efficiently
to get more billing later, but I really wanted to be one and done.  I
was and this made hand off to their Sysadmin when they did hire one
very smooth as everything that was running was recreatable and how it
was created was relatively easy to inspect which mostly removes the
'one systems person with everything in their head' single point of

I'd guess given the small size adding the automation layers probably
cost 20% of the project time -vs- just making it work by hand.  But
I'm confident it saved way more than that in transition costs and
ongoing operations costs when they expanded, and really who wants to
be just 5 servers in a closet?


More information about the OpenStack-operators mailing list