Open Stack

-----Original Message-----
From: Thomas Goirand <zigo at debian.org>
Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Date: April 18, 2016 at 17:21:36
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Subject:  Re: [openstack-dev] [release][requirements][packaging][summit] input needed on summit discussion about global requirements

> Hi Doug,
>  
> I very much welcome opening such a thread before the discussion at the
> summit, as often, sessions are too short. Taking the time to write
> things down first also helps having a more constructive discussion.
>  
> Before I reply to each individual message below, let me attempt to reply
> to the big picture seen in your etherpad. I was tempted to insert
> comments on each lines of it, but I'm not sure how this would be
> received, and probably it's best to attempt to reply more globally.
>  
> From what I understand, the biggesgt problems you're trying to solve is
> that managing the global-reqs is really time consuming from the release
> team point of view, and especially its propagation to individual
> projects. There's IMO many things that we could do to improve the
> situation, which would be acceptable from the package maintainers point
> of view.
>  
> First of all, from what I could see in the etherpad, I see a lot of
> release work which I consider not useful for anyone: not for downstream
> distros, not upstream projects. Mostly, the propagation of the
> global-requirements.txt to each and every individual Python library or
> service *for OpenStack maintained libs* could be reviewed. Because 1/
> distros will always package the highest version available in
> upper-constraints.txt, and 2/ it doesn't really reflect a reality. As
> you pointed out, project A may need a new feature from lib X, but
> project B wont care. I strongly believe that we should leave lower
> boundaries as a responsibility of individual projects. What important
> though, is to make sure that the highest version released does work,
> because that's what we will effectively package.
>  
> What we can then later on do, at the distribution level, is artificially
> set the lower bounds of versions to whatever we've just uploaded for a
> given release of OpenStack. In fact, I've been doing this a lot already.
> For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was
> never anything in the between. Therefore, doing a dependency like:
>  
> Depends: python-eventlet (>= 0.18.3)
>  
> makes no sense, and I always pushed:
>  
> Depends: python-eventlet (>= 0.18.4)
>  
> as this reflects the reality of distros.
>  
> If we generalize this concept, then I could push the minimum version of
> all oslo libs into every single package for a given version of OpenStack.
>  
> What is a lot more annoying though, is for packages which I do not
> control directly, and which are used by many other non-OpenStack
> packages inside the distribution. For example, Django, SQLAlchemy or
> jQuery, to only name a few.
>  
> I have absolutely no problem upping the lower bounds for all of
> OpenStack components aggressively. We don't have gate jobs for the lower
> bounds of our requirements. If we decide that it becomes the norm, I can
> generalize and push for doing this even more. For example, after pushing
> the update of an oslo lib B version X, I could push such requirements
> everywhere, which in fact, would be a good thing (as this would trigger
> rebuilds and recheck of all unit tests). Though, all of this would
> benefit from a lot of automation and checks.
>  
> On your etherpad, you wrote:
>  
> "During the lead-up to preparing the final releases, one of the tracking
> tasks we have is to ensure all projects have synced their global
> requirements updates. This is another area where we could reduce the
> burden on the release team."
>  
> Well, don't bother, this doesn't reflect a reality anyway (ie: maybe
> service X can use an older version of oslo.utils), so that's not really
> helpful in any way.
>  
> You also wrote:
>  
> "Current ranges in global-requirements are large but most projects do
> not actively test the oldest supported version (or other versions in
> between) meaning that the requirement might result in broken packages."
>  
> Yeah, that's truth, I've seen this and reported a few bugs (the last I
> have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still
> very useful hints for package maintainers *for 3rd party libs* (as I
> wrote, it's less important for OpenStack components). We have a few
> breakage here and there, but they are hopefully fixes.
>  
> Though having a single version that projects are allowed to test with is
> super important, so we can check everything can work together. IMO,
> that's the part which should absolutely not change. Dropping that is
> like opening a Pandora box. Relying on containers and/or venv will
> unfortunately, not work, from my stand point.
>  
> The general rule for a distribution is that the highest version always
> win, otherwise, it's never maintainable (for security and bug fixes). It
> should be the case for *any program*, not even just any OpenStack
> components. There's never a case where it's ok to use something older,
> just because it feels like less work to do. This type of "laziness"
> leads to very dangerous outcomes, always.
>  
> Though I don't see any issue with a project willing to keep backward
> compatibility with a lower version than what other project do. In fact,
> that's highly desirable to always try to remain compatible with lower
> versions as much as possible, *if* it doesn't bring too much burden
> (think: we still have loads of Python 2.6 stuff like discover, argparse
> and such that needs to be cleaned-up).
>  
> For the projects who aren't following the release cycles, it's ok if
> they bump their lower bounds at the condition that they keep what's in
> the upper-constraints.txt, so that we can release them together with the
> rest of OpenStack. For example, it's ok to require the latest oslo.utils
> 3.8.0 for Ironic 5.1.0.
>  
> I also see no reason to completely freeze versions to the point were it
> is impossible fix bugs in the versionnings. I've seen that multiple
> times. At the distro level, we do address the issue anyway, so why
> bother? The only thing that counts for us, is to be able to release
> everything together. A good example would be that Neutron needing SQLA
> >= 1.0.12: we refuse to fix it, but the issue is still there, and we
> have to fix it in distros.
>  
> Last, before I reply to each individual message: the attempt from Robert
> to test on lower bounds is really something that should be pushed
> forward. I would help a lot to actually know what the reality is, rather
> than doing second-guess and artificially set lower bounds for versions
> which we think may probably work.
>  
> On 04/18/2016 02:22 PM, Chris Dent wrote:
> > targeting an
> > all-in-one install seems only to serve the purposes of all-in-one rpm-
> > or deb-based installations.
>  
> Though that's what most everyone consumes. Or are you proposing that we
> completely stop publishing OpenStack in distros?
>  
> Remember that in distros, there's only a single version of a library at
> any given time, at the exception of transitions (yes, in Red Hat it's
> technically possible to install multiple versions, but the policy
> strongly advocates against this).
>  
> Also, all-in-one is what I use in Debian in my CI, to make sure that
> everyone works together, with whatever is uploaded (see the
> openstack-deploy package in Debian).
>  
> > Many (most?) people won't be doing those kinds of installations.
>  
> Most users are consuming packages from distributions. Also, if you're
> using containers, probably you will also prefer using these packages to
> build your containers: that's the most easy, safe and fast way to build
> your containers.
>  
> > If all-in-one installations are important to the rpm- and deb- based
> > distributions then _they_ should be resolving the dependency issues
> > local to their own infrastructure
>  
> Who is "they"? Well, approximately 1 person full time for Debian, 1 for
> Ubuntu if you combine Corey and David (and Debian and Ubuntu
> dependencies are worked on together in Debian), so that's 2 full time
> for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is
> still accurate.
>  
> So "we" wont handle it, even if "we" care, because we're already
> understaffed.
>  
> > (or realizing that it is too painful and start
> > containerizing or otherwise as well).
>  
> Distributions don't package containers, we offer packages that you can
> use to build them. The way you deploy is IMO orthogonal to packaging.
>  
> > I think making these changes will help to improve and strengthen the
> > boundaries and contracts between services. If not technically then
> > at least socially, in the sense that the negotiations that people
> > make to get things to work are about what actually matters in their
> > services, not unwinding python dependencies and the like.
>  
> Yes, it simplifies the life of developers. But it's going to be hell on
> earth for production if we discover a serious vulnerability in a
> component, which will be deployed N times, with N versions. Are we then
> going to provide N patches? The vulnerability management team is also
> understaffed...
>  
> > One challenge I've run into the past is when devstack
> > plugin A has made an assumption about having access to a python
> > script provided by devstack plugin B, but it's not on $PATH or its
> > dependencies are not in the site-packages visible to the current
> > context. The solution here is to use full paths _into_ virtenvs.
>  
> Please take a step back. Devstack and virtualenv are for development,
> not for production. That's not, and should not become, our target.
> Applying the reasoning you have there will *not* work for distributions.
> Hopefully, that's what Doug is proposing, as he would still like things
> to work for distros (right?).
>  
> On 04/18/2016 03:24 PM, Hayes, Graham wrote:
> > It is also worth noting that according to the OpenStack User Survey
> > [0] 56% of deployments use "Unmodifed packages from the operating
> > system".
>  
> It is my believe (and hope, as this would mean more users) that some
> OpenStack users don't even know about this survey, and just install
> what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed
> packages" is the answer. Maybe some are deploying "slightly modified
> packages, but mostly what's upstream", which would bring the score even
> higher.
>  
> On 04/18/2016 07:33 PM, Doug Hellmann wrote:
> > Can you truly not imagine any other useful way to package OpenStack
> > other than individual packages with shared dependencies that would
> > be acceptable?
>  
> I could imagine that, but that's not at all what OpenStack is currently.
> Currently, each and every service is using Oslo, clients, and so on,
> plus the glue libs like os-brick, Castellan, you name it.
>  
> Or did I misunderstood your sentence above?
>  
> On 04/18/2016 07:49 PM, Sean Dague wrote:
> > A lot of distros specifically have policies against this kind of
> > bundling as well, because of security issues like this (which was so
> > very bad) - http://www.zlib.net/advisory-2002-03-11.txt
>  
> Gentoo is probably a special case (because building from source and so
> on), but I'd say that *ALL* distro do have such policy in place (at
> least, Red Hat, Debian, Ubuntu, SuSE...).
>  
> On 04/18/2016 08:30 PM, Doug Hellmann wrote:
> > Our upstream
> > solution can be pretty light-weight, and that leaves room for
> > downstream folks to make different choices.
>  
> I'm not sure what choice we'd have, but to face loads of issues and
> finally give-up packaging OpenStack if we have python module versions
> that are conflicting. Hopefully, your proposal wont allow that?
>  
> On 04/18/2016 08:40 PM, Doug Hellmann wrote:
> > Some folks seem to be conflating "upstream wants to stop worrying
> > about co-installability" with "upstream wants us to bundle
> > dependencies". That's not what I'm proposing.
>  
> Well, if we need multiple versions of the same Python module, what is
> the solution in production then?
>  
> > Another alternate that might work is for downstream folks to do
> > their own dependency management.
>  
> I fail to understand what you think distros could do to fix the
> situation of conflicting versions, other than bundling, which isn't
> acceptable. Could you explain what you mean by "downstream folks to do
> their own dependency management"?
>  
> > That's less optimal, and may not
> > change the outcome for Gentoo or Debian, where the downstream teams
> > are small (1 person each, I think?).
>  
> You can also add "Ubuntu" in the list here, as absolutely all OpenStack
> dependencies are maintained mostly by me, within Debian, and then later

"absolutely all" of OpenStack's dependencies are not maintained by you in Debian. A significant number are maintained by the DPMT (Debian Python Modules Team). The large majority are maintained by you, but not "absolutely all".

> synced to Ubuntu, with a bit of help from Ubuntu folks (but they are
> mostly busy with Canonical own products, and I'm the person doing most
> of the work, still). This is slowly shifting to a more spread workload,
> but unfortunately, we're not there yet.
>  
> As for Red Hat / RDO, Haikel told me they are "2 and a half", so
> basically, the situation there (unless it changed) is approximately the
> same amount of burnout people.
>  
> > I started the discussion to solicit ideas, but I would prefer to
> > avoid proposing what downstream should do because (a) I'm sure
> > folks want options and (b) I'm sure I'm not the person to identify
> > those options.
>  
> Well, I'm doing the packaging, and I'm not sure if we have any option
> either, if 2 python dependencies are conflicting.
>  
> On 04/18/2016 09:10 PM, Jeremy Stanley wrote:
> > On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote:
> >> Ya, I'd be happy to work more with upstream. I already review the
> >> stable-reqs updates and watch them for the stable branches I package
> >> for. Not sure what else is needed.
> >
> > Reviewing the master branch openstack/requirements repository
> > changes (to make sure deps being added are going to be sane things
> > for someone in your distro to maintain packages of in the long term)
> > would also make sense.
>  
> I've done some of that, and used to be reactive on it. But mostly, the
> way it's been done so far is very good (thanks to a lot of folks working
> on it), and it's taking too much of my time to review them. So I just
> pick it up before packaging a (beta-)version of OpenStack.
>  
> > To be clear, "Moving the burden to packagers" is not the only option
> > available to us. I've proposed one option for eliminating the issue,
> > which has some benefits for us upstream but obviously introduces
> > some other issues we would need to resolve. Another option is for
> > more people to get involved in managing the dependency list. Some
> > (most? all?) of those new people may come from distros, and sharing
> > the effort among them would make it easier than each of them doing
> > all of the work individually. Sort of like an open source project.
>  
> There's 2 options I see to solve the "people reviewing the
> global-requirements.txt changes aren't enough".
>  
> 1/ talk to my boss and the folks from Canonical (I can't tell for the
> RPM world) to allocate more staff for packaging (yes, I'm serious here!
> I can get you in touch...).
>  
> 2/ get a lot of dependencies out. Yes, let's really do that! We're
> constantly adding more, I've seen only very few going away. Currently
> the global-requirements.txt for stable/mitaka is 377 lines long. If you
> grep out "client", "oslo" and "os-", "openstack", comments, empty lines,
> and a few others, there's about 260 dependencies left. That's insane at
> multiple levels. When are we going to see a movement to clean this mess,
> and define real OpenStack standards?
>  
> Cheers,
>  
> Thomas Goirand (zigo)
>  
>  
> __________________________________________________________________________  
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe  
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>  

--  
Ian Cordasco