Open Stack

Excerpts from James Slagle's message of 2014-01-07 15:03:33 -0800:
> On Tue, Jan 7, 2014 at 5:18 PM, Clint Byrum <clint at fewbar.com> wrote:
> > Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800:
> >> On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum <clint at fewbar.com> wrote:
> >> > What would be the benefit of using packages?
> >>
> >> We're building images on top of different distributions of Linux.
> >> Those distributions themselves offer packaged and supported OpenStack
> >> components.  So, one benefit is that you'd be using what's "blessed"
> >> by your distro if you chose to.  I think that's a farily common way
> >> people are going to be used to installing components. The OpenStack
> >> Installation guide says to install from packages, fwiw.
> >>
> >
> > Indeed, this is how many people deploy traditional applications.
> >
> > However, what we're doing is intended to be a real, consumable deployment
> > of OpenStack. Specifically one that is in the gate and scales out to
> > any reasonable production load necessary.
> >
> > One problem with scaling out to many nodes is that the traditional
> > application deployment patterns introduce too much entropy. This is really
> > hard to patch out later. We are designing the software distribution system
> > to avoid those problems from the beginning. Packages do the opposite,
> > and encourage entropy by promising to try and update software with
> > minimal invasion, thus enabling users to introduce one-off machines.
> 
> This sounds more like an argument for a systems management approach
> vs. installation.  I realize there's a big relation there.  However, I
> don't think just using an image based system makes the entropy problem
> go away.  At scale of 10,000 nodes (or more), you could easily have a
> proliferation of images both that you've built and that you've
> deployed.  You're not going to update everything at once.  Nor, do I
> think you would only ever have 2 images running, for your latest
> version N, and N-1..  You're going to have several different deployed
> images running to account for hardware differences, updates that have
> not yet been applied, migrations, etc.  My point is, the entropy
> problem does not go away.  Ergo, it's not introduced by packages.
> 
> Certainly it could be made worse by managing packages and their
> updates by hand across 10,000 nodes.   But again, I don't think anyone
> does that, they use a systems management tool that exists to
> discourage drift and help with such entropy.
> 
> Likewise, you're not going to be calling "nova rebuild" 10,000 times
> manually when you want to do image based updates.  You'd likely have
> some tool (tuskar, something else, etc) that is going to manage that
> for you, and help keep any drift in what images you actually have
> deployed in check.
> 

Image proliferation is far easier to measure than package proliferation.
But, that is not really the point.

The point is that we have a tool for software distribution that fits into
our system management approach, and the distro packages do not take that
system management approach into account.

So if you can't use the distro packages as-is, I'm questioning what the
actual benefit of using them at all is.

> >
> >> > We've specifically avoided packages because they complect[1] configuration
> >> > and system state management with software delivery. The recent friction
> >> > we've seen with MySQL is an example where the packages are not actually
> >> > helping us, they're hurting us because they encode too much configuration
> >> > instead of just delivering binaries.
> >>
> >> We're trying to do something fairly specific with the read only /
> >> partition.  You're right, most packages aren't going to handle that
> >> well.  So, yes you have a point from that perspective.
> >>
> >
> > Readonly / is a really important feature of the deployment we're aiming
> > at. Doing it with packages is quite possible. My point in asking why
> > bother with packages is that when you have an entire image that has been
> > verified and is known to work, what advantage does having a package for
> > everything actually bring.
> 
> Because distro packages are known to work, and thus you get higher
> confidence from any image constructed from said packages.  At least, I
> would, as opposed to installing from source (or "from git" as you say
> below :).  It's the same reason I want to use a packaged kernel
> instead of compiling it from source.  The benefit of the package is
> not just in the compiling.  It's in the known good version and
> compatibility with other known good versions I want to use.
> 

I would disagree with "known to work". They are known to have been
tested at some level. But IMO "known to work" requires testing _with
your workload_.

Since you have to test your workload, why bother with the distro packages
when you can get the upstream software and testing suite directly.

> Am I going to implicitly trust any packages blindly or completely?  Of
> course not. But, there is some confidence there in that the distro has
> done some testing and said these versions are compatible, etc.
> 

I think that confidence is misplaced and unnecessary. We provide test
suites to users and we will encourage users to test their own things. I
imagine some will also ship packaged products based on TripleO that will
also be tested as a whole, not as individual packages.

> >
> >> However, there are many examples of when packages help you.
> >> Dependency resolution, version compatibility, methods of verification,
> >> knowing what's installed, etc.  I don't think that's really an
> >> argument or discussion worth having, because you either want to use
> >> packages or you want to build it all from source.  There are
> >> advantages/disadvantages to both methods, and what I'm proposing is
> >> that we support both methods, and not require everyone to only be able
> >> to install from source.
> >>
> >
> > "Install from source" is probably not the right way to put this. We're
> > installing the virtualenvs from tarballs downloaded from pypi. We're
> > also installing 99.9% python, so we're not really going "from source",
> > we're just going "from git".
> 
> Yes, "from source" basically means "from git".  But, I fail to see the
> distinction you're making in this context. Yes, the python source is
> for all intents the same as the executable/library that is used at
> runtime. Why are you opposed to installing that "source" from a
> package vs a git repo?  What does using the git repo buy you?  If it's
> avoiding the added complexity of the package, I already pointed out
> some advantages to packaging as to why some people choose to use them
> and thus accept that complexity
> 

The distinction is just that there's not a big advantage of the binaries
being packaged separate from "source", that was all.

> > But anyway, I see your point and will capitulate that it is less weird
> > for people and thus may make the pill a little easier to swallow. But if
> > I could have it my way, I'd suggest that the packages be built to mirror
> > the structure of the element end-products as much as possible so that
> > they can be used with minimal change to elements.
> 
> I don't know to what degree that's possible.  However, I think most
> distros try to be as "upstream" friendly as possible.  In this
> context, if the OpenStack community says "this is how we recommend you
> install our stuff, and our reference is t-i-e", then I think packagers
> and distros are inclined to follow that closely as much as possible.
>

Perhaps Fedora is more liberal, but I doubt Debian policy is
going to allow default writing to /mnt/state or binaries in
/opt/stack/venvs/nova/bin. :)

> >
> >> > Perhaps those of us who have been involved a bit longer haven't done
> >> > a good job of communicating our reasons. I for one believe in the idea
> >> > that image based updates eliminate a lot of the entropy that comes along
> >> > with a package based updating system. For that reason alone I tend to
> >> > look at any packages that deliver configurable software as potentially
> >> > dangerous (note that I think they're wonderful for libraries, utilities,
> >> > and kernels. :)
> >>
> >> Using packages wouldn't prevent you from using the image based update
> >> mechanism.  Anecdotally, I think image based updates could be a bit
> >> heavy handed for something like picking up a quick security or bug fix
> >> or the like.  That would be a scenario where a package update could
> >> really be handy.  Especially if someone else (e.g., your distro) is
> >> maintaining the package for you.
> >>
> >> For this proposal though, I was only talking about installation of the
> >> components at image build time.
> >>
> >
> > The entire point of image based updates is that they are heavy handed.
> > The problem we're trying to solve is that you have a data center of (n)
> > machines and you don't want (n) unique sets of software,  where each
> > machine might have some hot fixes and not others. At 1000 machines it
> > becomes critical. At 10000 machines, the entropy matrix starts to get
> > scary.
> 
> I think I addressed this point earlier up.  But, using packages
> doesn't mean (n) unique sets of software.  And using images doesn't
> mean 1 unique set of software.  Certainly, we can come up with
> solutions that aim to make it such that entropy is not introduced
> accidentally.  And I think an image approach is a very good mechanism
> for that.  But, even that does not protect completely against a well
> meaning operator accidentally deploying the wrong thing, or making a
> change that is not reflected in the image.  Just like we can't protect
> against them logging in and running "yum update foo" inconsistently
> across their deployment.
> 

We can stop all of those things with or without packages. My point isn't
to say packages are harmful in an image-build-only context, it is to
say that I don't see the benefit.

I think you're basically saying they're worth complexity because somebody
else tests them for you. I disagree, but I definitely would understand
if people said I sound crazy and still want those packages.