Open Stack

Excerpts from Thomas Goirand's message of 2016-04-19 00:19:03 +0200:
> Hi Doug,
> 
> I very much welcome opening such a thread before the discussion at the
> summit, as often, sessions are too short. Taking the time to write
> things down first also helps having a more constructive discussion.

Thanks for your long and thoughtful response. :-)

> Before I reply to each individual message below, let me attempt to reply
> to the big picture seen in your etherpad. I was tempted to insert
> comments on each lines of it, but I'm not sure how this would be
> received, and probably it's best to attempt to reply more globally.
> 
> From what I understand, the biggesgt problems you're trying to solve is
> that managing the global-reqs is really time consuming from the release
> team point of view, and especially its propagation to individual
> projects. There's IMO many things that we could do to improve the
> situation, which would be acceptable from the package maintainers point
> of view.
> 
> First of all, from what I could see in the etherpad, I see a lot of
> release work which I consider not useful for anyone: not for downstream
> distros, not upstream projects. Mostly, the propagation of the
> global-requirements.txt to each and every individual Python library or
> service *for OpenStack maintained libs* could be reviewed. Because 1/
> distros will always package the highest version available in
> upper-constraints.txt, and 2/ it doesn't really reflect a reality. As
> you pointed out, project A may need a new feature from lib X, but
> project B wont care. I strongly believe that we should leave lower
> boundaries as a responsibility of individual projects. What important
> though, is to make sure that the highest version released does work,
> because that's what we will effectively package.
> 
> What we can then later on do, at the distribution level, is artificially
> set the lower bounds of versions to whatever we've just uploaded for a
> given release of OpenStack. In fact, I've been doing this a lot already.

That's great to know. We have the global list and the upper constraints
list, and both are intended to be used to signal to packaging folks what
we think ought to be used. I'm glad that signaling is working, and maybe
that means you're right that we don't need to sync the list absolutely,
just as a set of "compatible" ranges.

> For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was
> never anything in the between. Therefore, doing a dependency like:
> 
> Depends: python-eventlet (>= 0.18.3)
> 
> makes no sense, and I always pushed:
> 
> Depends: python-eventlet (>= 0.18.4)
> 
> as this reflects the reality of distros.
> 
> If we generalize this concept, then I could push the minimum version of
> all oslo libs into every single package for a given version of OpenStack.
> 
> What is a lot more annoying though, is for packages which I do not
> control directly, and which are used by many other non-OpenStack
> packages inside the distribution. For example, Django, SQLAlchemy or
> jQuery, to only name a few.
> 
> I have absolutely no problem upping the lower bounds for all of
> OpenStack components aggressively. We don't have gate jobs for the lower
> bounds of our requirements. If we decide that it becomes the norm, I can
> generalize and push for doing this even more. For example, after pushing
> the update of an oslo lib B version X, I could push such requirements
> everywhere, which in fact, would be a good thing (as this would trigger
> rebuilds and recheck of all unit tests). Though, all of this would
> benefit from a lot of automation and checks.

We don't generally raise the minimum version of a dependency unless we
actually need a new feature in that version. At least one reason for
this is to give packagers some flexibility, in case a slightly older
version of a lib works fine and is already packaged, for example.

> 
> On your etherpad, you wrote:
> 
> "During the lead-up to preparing the final releases, one of the tracking
> tasks we have is to ensure all projects have synced their global
> requirements updates. This is another area where we could reduce the
> burden on the release team."
> 
> Well, don't bother, this doesn't reflect a reality anyway (ie: maybe
> service X can use an older version of oslo.utils), so that's not really
> helpful in any way.

Right, that's why I proposed stopping the whole business with managing
the ranges. Your proposal seems to be somewhere in the middle between
doing what we do today (obsessively keeping in sync) and what I'm
proposing (abandoning any pretense of keeping in sync). Is that right?

> 
> You also wrote:
> 
> "Current ranges in global-requirements are large but most projects do
> not actively test the oldest supported version (or other versions in
> between) meaning that the requirement might result in broken packages."
> 
> Yeah, that's truth, I've seen this and reported a few bugs (the last I
> have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still
> very useful hints for package maintainers *for 3rd party libs* (as I
> wrote, it's less important for OpenStack components). We have a few
> breakage here and there, but they are hopefully fixes.
> 
> Though having a single version that projects are allowed to test with is
> super important, so we can check everything can work together. IMO,
> that's the part which should absolutely not change. Dropping that is
> like opening a Pandora box. Relying on containers and/or venv will
> unfortunately, not work, from my stand point.

Again, as I pointed out elsewhere in the thread, using some sort of
environment isolation tool in upstream CI testing does not imply that
the same or a similar tool needs to be used by distros for packaging.

> The general rule for a distribution is that the highest version always
> win, otherwise, it's never maintainable (for security and bug fixes). It
> should be the case for *any program*, not even just any OpenStack
> components. There's never a case where it's ok to use something older,
> just because it feels like less work to do. This type of "laziness"
> leads to very dangerous outcomes, always.
> 
> Though I don't see any issue with a project willing to keep backward
> compatibility with a lower version than what other project do. In fact,
> that's highly desirable to always try to remain compatible with lower
> versions as much as possible, *if* it doesn't bring too much burden
> (think: we still have loads of Python 2.6 stuff like discover, argparse
> and such that needs to be cleaned-up).

Aside: We should figure out a way to make a list of those things,
so we can work on the cleanup.

> For the projects who aren't following the release cycles, it's ok if
> they bump their lower bounds at the condition that they keep what's in
> the upper-constraints.txt, so that we can release them together with the
> rest of OpenStack. For example, it's ok to require the latest oslo.utils
> 3.8.0 for Ironic 5.1.0.

Most of our official projects *do* follow the release cycle. The
cycle-with-intermediary model explicitly follows the cycle by
preparing a final release around the same time as everyone else.

> I also see no reason to completely freeze versions to the point were it
> is impossible fix bugs in the versionnings. I've seen that multiple
> times. At the distro level, we do address the issue anyway, so why
> bother? The only thing that counts for us, is to be able to release
> everything together. A good example would be that Neutron needing SQLA
> >= 1.0.12: we refuse to fix it, but the issue is still there, and we
> have to fix it in distros.
> 
> Last, before I reply to each individual message: the attempt from Robert
> to test on lower bounds is really something that should be pushed
> forward. I would help a lot to actually know what the reality is, rather
> than doing second-guess and artificially set lower bounds for versions
> which we think may probably work.
> 
> On 04/18/2016 02:22 PM, Chris Dent wrote:
> > targeting an
> > all-in-one install seems only to serve the purposes of all-in-one rpm-
> > or deb-based installations.
> 
> Though that's what most everyone consumes. Or are you proposing that we
> completely stop publishing OpenStack in distros?

I'm certainly not proposing that. My proposal is point out the
economics of our current situation, which is that upstream we're
doing a lot of work that IMHO we don't need to do *for ourselves*.
I do see its usefulness, but it's not necessary for our own CI
testing.

> Remember that in distros, there's only a single version of a library at
> any given time, at the exception of transitions (yes, in Red Hat it's
> technically possible to install multiple versions, but the policy
> strongly advocates against this).
> 
> Also, all-in-one is what I use in Debian in my CI, to make sure that
> everyone works together, with whatever is uploaded (see the
> openstack-deploy package in Debian).
> 
> > Many (most?) people won't be doing those kinds of installations.
> 
> Most users are consuming packages from distributions. Also, if you're
> using containers, probably you will also prefer using these packages to
> build your containers: that's the most easy, safe and fast way to build
> your containers.
> 
> > If all-in-one installations are important to the rpm- and deb- based
> > distributions  then _they_ should be resolving the dependency issues
> > local to their own infrastructure
> 
> Who is "they"? Well, approximately 1 person full time for Debian, 1 for
> Ubuntu if you combine Corey and David (and Debian and Ubuntu
> dependencies are worked on together in Debian), so that's 2 full time
> for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is
> still accurate.
> 
> So "we" wont handle it, even if "we" care, because we're already
> understaffed.

Welcome to the club! :-) Seriously, we all are. That's why I'm looking
for opportunities to stop doing things that take a lot of human
attention.

> 
> > (or realizing that it is too painful and start
> > containerizing or otherwise as well).
> 
> Distributions don't package containers, we offer packages that you can
> use to build them. The way you deploy is IMO orthogonal to packaging.
> 
> > I think making these changes will help to improve and strengthen the
> > boundaries and contracts between services. If not technically then
> > at least socially, in the sense that the negotiations that people
> > make to get things to work are about what actually matters in their
> > services, not unwinding python dependencies and the like.
> 
> Yes, it simplifies the life of developers. But it's going to be hell on
> earth for production if we discover a serious vulnerability in a
> component, which will be deployed N times, with N versions. Are we then
> going to provide N patches? The vulnerability management team is also
> understaffed...
> 
> > One challenge I've run into the past is when devstack
> > plugin A has made an assumption about having access to a python
> > script provided by devstack plugin B, but it's not on $PATH or its
> > dependencies are not in the site-packages visible to the current
> > context. The solution here is to use full paths _into_ virtenvs.
> 
> Please take a step back. Devstack and virtualenv are for development,
> not for production. That's not, and should not become, our target.
> Applying the reasoning you have there will *not* work for distributions.
> Hopefully, that's what Doug is proposing, as he would still like things
> to work for distros (right?).
> 
> On 04/18/2016 03:24 PM, Hayes, Graham wrote:
> > It is also worth noting that according to the OpenStack User Survey
> > [0] 56% of deployments use "Unmodifed packages from the operating
> > system".
> 
> It is my believe (and hope, as this would mean more users) that some
> OpenStack users don't even know about this survey, and just install
> what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed
> packages" is the answer. Maybe some are deploying "slightly modified
> packages, but mostly what's upstream", which would bring the score even
> higher.
> 
> On 04/18/2016 07:33 PM, Doug Hellmann wrote:
> > Can you truly not imagine any other useful way to package OpenStack
> > other than individual packages with shared dependencies that would
> > be acceptable?
> 
> I could imagine that, but that's not at all what OpenStack is currently.
> Currently, each and every service is using Oslo, clients, and so on,
> plus the glue libs like os-brick, Castellan, you name it.
> 
> Or did I misunderstood your sentence above?

I was trying to say that if it's too much work for distros to manage
the dependencies individually, or by joining the review team, maybe
alternative packaging strategies *should* be examined. Based on the
responses from distros so far, I doubt that any of you are ready
for that today, which leaves us still needing to address the problem
of how much effort we're putting into this today.

> 
> On 04/18/2016 07:49 PM, Sean Dague wrote:
> > A lot of distros specifically have policies against this kind of
> > bundling as well, because of security issues like this (which was so
> > very bad) - http://www.zlib.net/advisory-2002-03-11.txt
> 
> Gentoo is probably a special case (because building from source and so
> on), but I'd say that *ALL* distro do have such policy in place (at
> least, Red Hat, Debian, Ubuntu, SuSE...).
> 
> On 04/18/2016 08:30 PM, Doug Hellmann wrote:
> > Our upstream
> > solution can be pretty light-weight, and that leaves room for
> > downstream folks to make different choices.
> 
> I'm not sure what choice we'd have, but to face loads of issues and
> finally give-up packaging OpenStack if we have python module versions
> that are conflicting. Hopefully, your proposal wont allow that?

That's certainly not what I want. Giving up strikes me as an easy but
not particularly constructive suggestion.

> 
> On 04/18/2016 08:40 PM, Doug Hellmann wrote:
> > Some folks seem to be conflating "upstream wants to stop worrying
> > about co-installability" with "upstream wants us to bundle
> > dependencies". That's not what I'm proposing.
> 
> Well, if we need multiple versions of the same Python module, what is
> the solution in production then?

I'm not sure. We have a lot of diverse experience with deployments
in our community. I'm asking for input from those folks.

> 
> > Another alternate that might work is for downstream folks to do
> > their own dependency management.
> 
> I fail to understand what you think distros could do to fix the
> situation of conflicting versions, other than bundling, which isn't
> acceptable. Could you explain what you mean by "downstream folks to do
> their own dependency management"?

How do you ensure that a given version of a python lib you package
is compatible with independent tools written to use that lib that
you also want to include in your distro? Or for that matter, anything
written in any other language that supports shared libraries. What
would you do if OpenStack was not one big thing produced by one
team, but lots of medium-sized things produced in a distributed
fashion?

> 
> > That's less optimal, and may not
> > change the outcome for Gentoo or Debian, where the downstream teams
> > are small (1 person each, I think?).
> 
> You can also add "Ubuntu" in the list here, as absolutely all OpenStack
> dependencies are maintained mostly by me, within Debian, and then later
> synced to Ubuntu, with a bit of help from Ubuntu folks (but they are
> mostly busy with Canonical own products, and I'm the person doing most
> of the work, still). This is slowly shifting to a more spread workload,
> but unfortunately, we're not there yet.
> 
> As for Red Hat / RDO, Haikel told me they are "2 and a half", so
> basically, the situation there (unless it changed) is approximately the
> same amount of burnout people.

Yes, clearly that's not enough resources to do this job. And yet few
other companies are contributing the resources, either. If we want the
job done, we either need to make it easier or find more people.

> 
> > I started the discussion to solicit ideas, but I would prefer to
> > avoid proposing what downstream should do because (a) I'm sure
> > folks want options and (b) I'm sure I'm not the person to identify
> > those options.
> 
> Well, I'm doing the packaging, and I'm not sure if we have any option
> either, if 2 python dependencies are conflicting.

Surely OpenStack is not the only time when that situation arises?

> 
> On 04/18/2016 09:10 PM, Jeremy Stanley wrote:
> > On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote:
> >> Ya, I'd be happy to work more with upstream.  I already review the
> >> stable-reqs updates and watch them for the stable branches I package
> >> for.  Not sure what else is needed.
> >
> > Reviewing the master branch openstack/requirements repository
> > changes (to make sure deps being added are going to be sane things
> > for someone in your distro to maintain packages of in the long term)
> > would also make sense.
> 
> I've done some of that, and used to be reactive on it. But mostly, the
> way it's been done so far is very good (thanks to a lot of folks working
> on it), and it's taking too much of my time to review them. So I just
> pick it up before packaging a (beta-)version of OpenStack.

I think you've made my point here. Reviewing these changes *is*
time consuming.

> 
> > To be clear, "Moving the burden to packagers" is not the only option
> > available to us. I've proposed one option for eliminating the issue,
> > which has some benefits for us upstream but obviously introduces
> > some other issues we would need to resolve. Another option is for
> > more people to get involved in managing the dependency list. Some
> > (most? all?) of those new people may come from distros, and sharing
> > the effort among them would make it easier than each of them doing
> > all of the work individually. Sort of like an open source project.
> 
> There's 2 options I see to solve the "people reviewing the
> global-requirements.txt changes aren't enough".
> 
> 1/ talk to my boss and the folks from Canonical (I can't tell for the
> RPM world) to allocate more staff for packaging (yes, I'm serious here!
> I can get you in touch...).

If they'll have folks at the summit, that would be good. If not, we'll
see how the discussion there goes and then see what we need to do.

> 
> 2/ get a lot of dependencies out. Yes, let's really do that! We're
> constantly adding more, I've seen only very few going away. Currently
> the global-requirements.txt for stable/mitaka is 377 lines long. If you
> grep out "client", "oslo" and "os-", "openstack", comments, empty lines,
> and a few others, there's about 260 dependencies left. That's insane at
> multiple levels. When are we going to see a movement to clean this mess,
> and define real OpenStack standards?

That's a good question, too, and we should have a look at that list at
some point no matter what the outcome of this discussion. That said, as
we continue to expand into more and more areas with new projects, the
list isn't going to stay completely stable, so we need to build a system
capable of handling it.

Doug

> 
> Cheers,
> 
> Thomas Goirand (zigo)
>