Open Stack

On Tue, Jan 7, 2014 at 5:18 PM, Clint Byrum <clint at fewbar.com> wrote:
> Excerpts from James Slagle's message of 2014-01-07 12:53:57 -0800:
>> On Tue, Jan 7, 2014 at 3:23 PM, Clint Byrum <clint at fewbar.com> wrote:
>> > What would be the benefit of using packages?
>>
>> We're building images on top of different distributions of Linux.
>> Those distributions themselves offer packaged and supported OpenStack
>> components.  So, one benefit is that you'd be using what's "blessed"
>> by your distro if you chose to.  I think that's a farily common way
>> people are going to be used to installing components. The OpenStack
>> Installation guide says to install from packages, fwiw.
>>
>
> Indeed, this is how many people deploy traditional applications.
>
> However, what we're doing is intended to be a real, consumable deployment
> of OpenStack. Specifically one that is in the gate and scales out to
> any reasonable production load necessary.
>
> One problem with scaling out to many nodes is that the traditional
> application deployment patterns introduce too much entropy. This is really
> hard to patch out later. We are designing the software distribution system
> to avoid those problems from the beginning. Packages do the opposite,
> and encourage entropy by promising to try and update software with
> minimal invasion, thus enabling users to introduce one-off machines.

This sounds more like an argument for a systems management approach
vs. installation.  I realize there's a big relation there.  However, I
don't think just using an image based system makes the entropy problem
go away.  At scale of 10,000 nodes (or more), you could easily have a
proliferation of images both that you've built and that you've
deployed.  You're not going to update everything at once.  Nor, do I
think you would only ever have 2 images running, for your latest
version N, and N-1..  You're going to have several different deployed
images running to account for hardware differences, updates that have
not yet been applied, migrations, etc.  My point is, the entropy
problem does not go away.  Ergo, it's not introduced by packages.

Certainly it could be made worse by managing packages and their
updates by hand across 10,000 nodes.   But again, I don't think anyone
does that, they use a systems management tool that exists to
discourage drift and help with such entropy.

Likewise, you're not going to be calling "nova rebuild" 10,000 times
manually when you want to do image based updates.  You'd likely have
some tool (tuskar, something else, etc) that is going to manage that
for you, and help keep any drift in what images you actually have
deployed in check.

>
>> > We've specifically avoided packages because they complect[1] configuration
>> > and system state management with software delivery. The recent friction
>> > we've seen with MySQL is an example where the packages are not actually
>> > helping us, they're hurting us because they encode too much configuration
>> > instead of just delivering binaries.
>>
>> We're trying to do something fairly specific with the read only /
>> partition.  You're right, most packages aren't going to handle that
>> well.  So, yes you have a point from that perspective.
>>
>
> Readonly / is a really important feature of the deployment we're aiming
> at. Doing it with packages is quite possible. My point in asking why
> bother with packages is that when you have an entire image that has been
> verified and is known to work, what advantage does having a package for
> everything actually bring.

Because distro packages are known to work, and thus you get higher
confidence from any image constructed from said packages.  At least, I
would, as opposed to installing from source (or "from git" as you say
below :).  It's the same reason I want to use a packaged kernel
instead of compiling it from source.  The benefit of the package is
not just in the compiling.  It's in the known good version and
compatibility with other known good versions I want to use.

Am I going to implicitly trust any packages blindly or completely?  Of
course not. But, there is some confidence there in that the distro has
done some testing and said these versions are compatible, etc.

>
>> However, there are many examples of when packages help you.
>> Dependency resolution, version compatibility, methods of verification,
>> knowing what's installed, etc.  I don't think that's really an
>> argument or discussion worth having, because you either want to use
>> packages or you want to build it all from source.  There are
>> advantages/disadvantages to both methods, and what I'm proposing is
>> that we support both methods, and not require everyone to only be able
>> to install from source.
>>
>
> "Install from source" is probably not the right way to put this. We're
> installing the virtualenvs from tarballs downloaded from pypi. We're
> also installing 99.9% python, so we're not really going "from source",
> we're just going "from git".

Yes, "from source" basically means "from git".  But, I fail to see the
distinction you're making in this context. Yes, the python source is
for all intents the same as the executable/library that is used at
runtime. Why are you opposed to installing that "source" from a
package vs a git repo?  What does using the git repo buy you?  If it's
avoiding the added complexity of the package, I already pointed out
some advantages to packaging as to why some people choose to use them
and thus accept that complexity

> But anyway, I see your point and will capitulate that it is less weird
> for people and thus may make the pill a little easier to swallow. But if
> I could have it my way, I'd suggest that the packages be built to mirror
> the structure of the element end-products as much as possible so that
> they can be used with minimal change to elements.

I don't know to what degree that's possible.  However, I think most
distros try to be as "upstream" friendly as possible.  In this
context, if the OpenStack community says "this is how we recommend you
install our stuff, and our reference is t-i-e", then I think packagers
and distros are inclined to follow that closely as much as possible.

>
>> > Perhaps those of us who have been involved a bit longer haven't done
>> > a good job of communicating our reasons. I for one believe in the idea
>> > that image based updates eliminate a lot of the entropy that comes along
>> > with a package based updating system. For that reason alone I tend to
>> > look at any packages that deliver configurable software as potentially
>> > dangerous (note that I think they're wonderful for libraries, utilities,
>> > and kernels. :)
>>
>> Using packages wouldn't prevent you from using the image based update
>> mechanism.  Anecdotally, I think image based updates could be a bit
>> heavy handed for something like picking up a quick security or bug fix
>> or the like.  That would be a scenario where a package update could
>> really be handy.  Especially if someone else (e.g., your distro) is
>> maintaining the package for you.
>>
>> For this proposal though, I was only talking about installation of the
>> components at image build time.
>>
>
> The entire point of image based updates is that they are heavy handed.
> The problem we're trying to solve is that you have a data center of (n)
> machines and you don't want (n) unique sets of software,  where each
> machine might have some hot fixes and not others. At 1000 machines it
> becomes critical. At 10000 machines, the entropy matrix starts to get
> scary.

I think I addressed this point earlier up.  But, using packages
doesn't mean (n) unique sets of software.  And using images doesn't
mean 1 unique set of software.  Certainly, we can come up with
solutions that aim to make it such that entropy is not introduced
accidentally.  And I think an image approach is a very good mechanism
for that.  But, even that does not protect completely against a well
meaning operator accidentally deploying the wrong thing, or making a
change that is not reflected in the image.  Just like we can't protect
against them logging in and running "yum update foo" inconsistently
across their deployment.

Just to clarify, I'm not arguing against the image based update
approach.  I'm proposing a way to install the OpenStack components
from packages at image build time.

-- 
-- James Slagle
--