Open Stack

Hi Doug,

I very much welcome opening such a thread before the discussion at the
summit, as often, sessions are too short. Taking the time to write
things down first also helps having a more constructive discussion.

Before I reply to each individual message below, let me attempt to reply
to the big picture seen in your etherpad. I was tempted to insert
comments on each lines of it, but I'm not sure how this would be
received, and probably it's best to attempt to reply more globally.

>From what I understand, the biggesgt problems you're trying to solve is
that managing the global-reqs is really time consuming from the release
team point of view, and especially its propagation to individual
projects. There's IMO many things that we could do to improve the
situation, which would be acceptable from the package maintainers point
of view.

First of all, from what I could see in the etherpad, I see a lot of
release work which I consider not useful for anyone: not for downstream
distros, not upstream projects. Mostly, the propagation of the
global-requirements.txt to each and every individual Python library or
service *for OpenStack maintained libs* could be reviewed. Because 1/
distros will always package the highest version available in
upper-constraints.txt, and 2/ it doesn't really reflect a reality. As
you pointed out, project A may need a new feature from lib X, but
project B wont care. I strongly believe that we should leave lower
boundaries as a responsibility of individual projects. What important
though, is to make sure that the highest version released does work,
because that's what we will effectively package.

What we can then later on do, at the distribution level, is artificially
set the lower bounds of versions to whatever we've just uploaded for a
given release of OpenStack. In fact, I've been doing this a lot already.
For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was
never anything in the between. Therefore, doing a dependency like:

Depends: python-eventlet (>= 0.18.3)

makes no sense, and I always pushed:

Depends: python-eventlet (>= 0.18.4)

as this reflects the reality of distros.

If we generalize this concept, then I could push the minimum version of
all oslo libs into every single package for a given version of OpenStack.

What is a lot more annoying though, is for packages which I do not
control directly, and which are used by many other non-OpenStack
packages inside the distribution. For example, Django, SQLAlchemy or
jQuery, to only name a few.

I have absolutely no problem upping the lower bounds for all of
OpenStack components aggressively. We don't have gate jobs for the lower
bounds of our requirements. If we decide that it becomes the norm, I can
generalize and push for doing this even more. For example, after pushing
the update of an oslo lib B version X, I could push such requirements
everywhere, which in fact, would be a good thing (as this would trigger
rebuilds and recheck of all unit tests). Though, all of this would
benefit from a lot of automation and checks.

On your etherpad, you wrote:

"During the lead-up to preparing the final releases, one of the tracking
tasks we have is to ensure all projects have synced their global
requirements updates. This is another area where we could reduce the
burden on the release team."

Well, don't bother, this doesn't reflect a reality anyway (ie: maybe
service X can use an older version of oslo.utils), so that's not really
helpful in any way.

You also wrote:

"Current ranges in global-requirements are large but most projects do
not actively test the oldest supported version (or other versions in
between) meaning that the requirement might result in broken packages."

Yeah, that's truth, I've seen this and reported a few bugs (the last I
have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still
very useful hints for package maintainers *for 3rd party libs* (as I
wrote, it's less important for OpenStack components). We have a few
breakage here and there, but they are hopefully fixes.

Though having a single version that projects are allowed to test with is
super important, so we can check everything can work together. IMO,
that's the part which should absolutely not change. Dropping that is
like opening a Pandora box. Relying on containers and/or venv will
unfortunately, not work, from my stand point.

The general rule for a distribution is that the highest version always
win, otherwise, it's never maintainable (for security and bug fixes). It
should be the case for *any program*, not even just any OpenStack
components. There's never a case where it's ok to use something older,
just because it feels like less work to do. This type of "laziness"
leads to very dangerous outcomes, always.

Though I don't see any issue with a project willing to keep backward
compatibility with a lower version than what other project do. In fact,
that's highly desirable to always try to remain compatible with lower
versions as much as possible, *if* it doesn't bring too much burden
(think: we still have loads of Python 2.6 stuff like discover, argparse
and such that needs to be cleaned-up).

For the projects who aren't following the release cycles, it's ok if
they bump their lower bounds at the condition that they keep what's in
the upper-constraints.txt, so that we can release them together with the
rest of OpenStack. For example, it's ok to require the latest oslo.utils
3.8.0 for Ironic 5.1.0.

I also see no reason to completely freeze versions to the point were it
is impossible fix bugs in the versionnings. I've seen that multiple
times. At the distro level, we do address the issue anyway, so why
bother? The only thing that counts for us, is to be able to release
everything together. A good example would be that Neutron needing SQLA
>= 1.0.12: we refuse to fix it, but the issue is still there, and we
have to fix it in distros.

Last, before I reply to each individual message: the attempt from Robert
to test on lower bounds is really something that should be pushed
forward. I would help a lot to actually know what the reality is, rather
than doing second-guess and artificially set lower bounds for versions
which we think may probably work.

On 04/18/2016 02:22 PM, Chris Dent wrote:
> targeting an
> all-in-one install seems only to serve the purposes of all-in-one rpm-
> or deb-based installations.

Though that's what most everyone consumes. Or are you proposing that we
completely stop publishing OpenStack in distros?

Remember that in distros, there's only a single version of a library at
any given time, at the exception of transitions (yes, in Red Hat it's
technically possible to install multiple versions, but the policy
strongly advocates against this).

Also, all-in-one is what I use in Debian in my CI, to make sure that
everyone works together, with whatever is uploaded (see the
openstack-deploy package in Debian).

> Many (most?) people won't be doing those kinds of installations.

Most users are consuming packages from distributions. Also, if you're
using containers, probably you will also prefer using these packages to
build your containers: that's the most easy, safe and fast way to build
your containers.

> If all-in-one installations are important to the rpm- and deb- based
> distributions  then _they_ should be resolving the dependency issues
> local to their own infrastructure

Who is "they"? Well, approximately 1 person full time for Debian, 1 for
Ubuntu if you combine Corey and David (and Debian and Ubuntu
dependencies are worked on together in Debian), so that's 2 full time
for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is
still accurate.

So "we" wont handle it, even if "we" care, because we're already
understaffed.

> (or realizing that it is too painful and start
> containerizing or otherwise as well).

Distributions don't package containers, we offer packages that you can
use to build them. The way you deploy is IMO orthogonal to packaging.

> I think making these changes will help to improve and strengthen the
> boundaries and contracts between services. If not technically then
> at least socially, in the sense that the negotiations that people
> make to get things to work are about what actually matters in their
> services, not unwinding python dependencies and the like.

Yes, it simplifies the life of developers. But it's going to be hell on
earth for production if we discover a serious vulnerability in a
component, which will be deployed N times, with N versions. Are we then
going to provide N patches? The vulnerability management team is also
understaffed...

> One challenge I've run into the past is when devstack
> plugin A has made an assumption about having access to a python
> script provided by devstack plugin B, but it's not on $PATH or its
> dependencies are not in the site-packages visible to the current
> context. The solution here is to use full paths _into_ virtenvs.

Please take a step back. Devstack and virtualenv are for development,
not for production. That's not, and should not become, our target.
Applying the reasoning you have there will *not* work for distributions.
Hopefully, that's what Doug is proposing, as he would still like things
to work for distros (right?).

On 04/18/2016 03:24 PM, Hayes, Graham wrote:
> It is also worth noting that according to the OpenStack User Survey
> [0] 56% of deployments use "Unmodifed packages from the operating
> system".

It is my believe (and hope, as this would mean more users) that some
OpenStack users don't even know about this survey, and just install
what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed
packages" is the answer. Maybe some are deploying "slightly modified
packages, but mostly what's upstream", which would bring the score even
higher.

On 04/18/2016 07:33 PM, Doug Hellmann wrote:
> Can you truly not imagine any other useful way to package OpenStack
> other than individual packages with shared dependencies that would
> be acceptable?

I could imagine that, but that's not at all what OpenStack is currently.
Currently, each and every service is using Oslo, clients, and so on,
plus the glue libs like os-brick, Castellan, you name it.

Or did I misunderstood your sentence above?

On 04/18/2016 07:49 PM, Sean Dague wrote:
> A lot of distros specifically have policies against this kind of
> bundling as well, because of security issues like this (which was so
> very bad) - http://www.zlib.net/advisory-2002-03-11.txt

Gentoo is probably a special case (because building from source and so
on), but I'd say that *ALL* distro do have such policy in place (at
least, Red Hat, Debian, Ubuntu, SuSE...).

On 04/18/2016 08:30 PM, Doug Hellmann wrote:
> Our upstream
> solution can be pretty light-weight, and that leaves room for
> downstream folks to make different choices.

I'm not sure what choice we'd have, but to face loads of issues and
finally give-up packaging OpenStack if we have python module versions
that are conflicting. Hopefully, your proposal wont allow that?

On 04/18/2016 08:40 PM, Doug Hellmann wrote:
> Some folks seem to be conflating "upstream wants to stop worrying
> about co-installability" with "upstream wants us to bundle
> dependencies". That's not what I'm proposing.

Well, if we need multiple versions of the same Python module, what is
the solution in production then?

> Another alternate that might work is for downstream folks to do
> their own dependency management.

I fail to understand what you think distros could do to fix the
situation of conflicting versions, other than bundling, which isn't
acceptable. Could you explain what you mean by "downstream folks to do
their own dependency management"?

> That's less optimal, and may not
> change the outcome for Gentoo or Debian, where the downstream teams
> are small (1 person each, I think?).

You can also add "Ubuntu" in the list here, as absolutely all OpenStack
dependencies are maintained mostly by me, within Debian, and then later
synced to Ubuntu, with a bit of help from Ubuntu folks (but they are
mostly busy with Canonical own products, and I'm the person doing most
of the work, still). This is slowly shifting to a more spread workload,
but unfortunately, we're not there yet.

As for Red Hat / RDO, Haikel told me they are "2 and a half", so
basically, the situation there (unless it changed) is approximately the
same amount of burnout people.

> I started the discussion to solicit ideas, but I would prefer to
> avoid proposing what downstream should do because (a) I'm sure
> folks want options and (b) I'm sure I'm not the person to identify
> those options.

Well, I'm doing the packaging, and I'm not sure if we have any option
either, if 2 python dependencies are conflicting.

On 04/18/2016 09:10 PM, Jeremy Stanley wrote:
> On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote:
>> Ya, I'd be happy to work more with upstream.  I already review the
>> stable-reqs updates and watch them for the stable branches I package
>> for.  Not sure what else is needed.
>
> Reviewing the master branch openstack/requirements repository
> changes (to make sure deps being added are going to be sane things
> for someone in your distro to maintain packages of in the long term)
> would also make sense.

I've done some of that, and used to be reactive on it. But mostly, the
way it's been done so far is very good (thanks to a lot of folks working
on it), and it's taking too much of my time to review them. So I just
pick it up before packaging a (beta-)version of OpenStack.

> To be clear, "Moving the burden to packagers" is not the only option
> available to us. I've proposed one option for eliminating the issue,
> which has some benefits for us upstream but obviously introduces
> some other issues we would need to resolve. Another option is for
> more people to get involved in managing the dependency list. Some
> (most? all?) of those new people may come from distros, and sharing
> the effort among them would make it easier than each of them doing
> all of the work individually. Sort of like an open source project.

There's 2 options I see to solve the "people reviewing the
global-requirements.txt changes aren't enough".

1/ talk to my boss and the folks from Canonical (I can't tell for the
RPM world) to allocate more staff for packaging (yes, I'm serious here!
I can get you in touch...).

2/ get a lot of dependencies out. Yes, let's really do that! We're
constantly adding more, I've seen only very few going away. Currently
the global-requirements.txt for stable/mitaka is 377 lines long. If you
grep out "client", "oslo" and "os-", "openstack", comments, empty lines,
and a few others, there's about 260 dependencies left. That's insane at
multiple levels. When are we going to see a movement to clean this mess,
and define real OpenStack standards?

Cheers,

Thomas Goirand (zigo)