Open Stack

Thomas, you've made a lot of great points here about the challenges
faced by packaging and packagers of OpenStack. They are valid,
important and need to be considered. I don't want to discount the
work you do because it is awesome, important and I'm well aware that
it can be a huge pain in the ass.

We also, however, need to consider what the future might look like and
at least for some people and situations, the future does not involve
debs or rpms of OpenStack: packages and distributions are more trouble
than they are worth when you want to be managing your infrastructure
across multiple, isolated environments. In that case you want
puppet, ansible, chef, docker, k8s, etc.

Sure all those things _can_ use packages to do their install but for
at least _some_ people that's redundant: deploy from a tag, branch,
commit or head of a repo.

That's for _some_ people. For other people packages are great.

Keep in mind that I'm presenting my own opinion here. I'm quite sure
it is different from, for example, Doug's. It's easy to conflate
arguments such that they appear the same. My position is radical (in
the sense that it is seeking to resolve root causes and institute
fundamental change), I suspect Doug's is considerably more moderate
and evolutionary.

More below.


On Tue, 19 Apr 2016, Thomas Goirand wrote:
> On 04/18/2016 02:22 PM, Chris Dent wrote:
>> targeting an
>> all-in-one install seems only to serve the purposes of all-in-one rpm-
>> or deb-based installations.
>
> Though that's what most everyone consumes. Or are you proposing that we
> completely stop publishing OpenStack in distros?

I'm not really making a proposal. I'm presenting my thoughts on the
matter to participate, by proxy, in the conversations that will
happen at summit. I'm hoping that a proposal that makes everyone
happy will come out of that. It just so happens that my position on
the matter is probably at one of the extremes. Do I expect it all to
happen like I want? No. Do I hope that my concerns will be
integrated in the discussion? Yes.

> Remember that in distros, there's only a single version of a library at
> any given time, at the exception of transitions (yes, in Red Hat it's
> technically possible to install multiple versions, but the policy
> strongly advocates against this).

Yes, which is part of why distros and packaging are limiting in the
cloud environment. When there is a security issue or other bug we don't
want to update those libraries via packages, nor update those libraries
across a bunch of different containers or what not.

We simply want to destroy the deployment and redeploy. With new
stuff. We want to do that without co-dependencies.

In other words, packaging of OpenStack into rpms and debs is a short
branch on the tree of life.

> Also, all-in-one is what I use in Debian in my CI, to make sure that
> everyone works together, with whatever is uploaded (see the
> openstack-deploy package in Debian).

Exactly, because this is what is needed to confirm one of your
requirements (that everything works together). Not everyone has that
requirement.

>> Many (most?) people won't be doing those kinds of installations.
>
> Most users are consuming packages from distributions. Also, if you're
> using containers, probably you will also prefer using these packages to
> build your containers: that's the most easy, safe and fast way to build
> your containers.

I predict that that is not going to last.

>> If all-in-one installations are important to the rpm- and deb- based
>> distributions  then _they_ should be resolving the dependency issues
>> local to their own infrastructure
>
> Who is "they"? Well, approximately 1 person full time for Debian, 1 for
> Ubuntu if you combine Corey and David (and Debian and Ubuntu
> dependencies are worked on together in Debian), so that's 2 full time
> for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is
> still accurate.

Yes, this is a significant issue and I think is one of the very
complicated aspects of the strange economics of OpenStack. It's been
clear from the start that the amount of people involved at the
distro level has been far too low to satisfy the requirements of the
users of those distributions. Far.

However, that problem is, I think, orthogonal to the question of
effectively creating OpenStack at the upstream (pre-packaging) level.

> So "we" wont handle it, even if "we" care, because we're already
> understaffed.

I agree. That's a problem for employers of the packagers to solve,
not OpenStack at large. You could make a pretty strong argument, of
course, that OpenStack at large _is_ those employers. That's
unfortunate and isn't the reality we should be aspiring to.

>> I think making these changes will help to improve and strengthen the
>> boundaries and contracts between services. If not technically then
>> at least socially, in the sense that the negotiations that people
>> make to get things to work are about what actually matters in their
>> services, not unwinding python dependencies and the like.
>
> Yes, it simplifies the life of developers. But it's going to be hell on
> earth for production if we discover a serious vulnerability in a
> component, which will be deployed N times, with N versions. Are we then
> going to provide N patches? The vulnerability management team is also
> understaffed...

See above to see my thinking on this.

>> One challenge I've run into the past is when devstack
>> plugin A has made an assumption about having access to a python
>> script provided by devstack plugin B, but it's not on $PATH or its
>> dependencies are not in the site-packages visible to the current
>> context. The solution here is to use full paths _into_ virtenvs.
>
> Please take a step back. Devstack and virtualenv are for development,
> not for production. That's not, and should not become, our target.
> Applying the reasoning you have there will *not* work for distributions.
> Hopefully, that's what Doug is proposing, as he would still like things
> to work for distros (right?).

Yes, devstack is for development, but the problem that we're trying
to solve for here is that OpenStack _development_ is held back by
attempting to manage global requirements in the way that we do.

One of the biggest reasons we do that is co-installability. One of
the biggest reasons we target co-installability is for the sake of
the distros.

The problem with all that is that co-installation is _bad_. Don't do
that! If we recognize that a lot of these problems go away.

> 2/ get a lot of dependencies out. Yes, let's really do that! We're
> constantly adding more, I've seen only very few going away. Currently
> the global-requirements.txt for stable/mitaka is 377 lines long. If you
> grep out "client", "oslo" and "os-", "openstack", comments, empty lines,
> and a few others, there's about 260 dependencies left. That's insane at
> multiple levels. When are we going to see a movement to clean this mess,
> and define real OpenStack standards?

That list looks long because it is used for many many projects that
strive for co-installability. If each project has its own list, the
lists are, individually, much shorter.

If we want there to be a big-tent (a debate worth having) then we're
going to have loads of projects, loads of requirements and a
distinct lack of standards. We're going to have to evolve to cope
with that.

-- 
Chris Dent               (¨s¡ã¡õ¡ã)¨s¦à©ß©¥©ß            http://anticdent.org/
freenode: cdent                                         tw: @anticdent