Open Stack

On Tue, Apr 19, 2016 at 12:24 PM, Ian Cordasco <sigmavirus24 at gmail.com>
wrote:

>
>
> -----Original Message-----
> From: Thomas Goirand <zigo at debian.org>
> Reply: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev at lists.openstack.org>
> Date: April 18, 2016 at 17:21:36
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev at lists.openstack.org>
> Subject:  Re: [openstack-dev] [release][requirements][packaging][summit]
> input needed on summit discussion about global requirements
>
> > Hi Doug,
> >
> > I very much welcome opening such a thread before the discussion at the
> > summit, as often, sessions are too short. Taking the time to write
> > things down first also helps having a more constructive discussion.
> >
> > Before I reply to each individual message below, let me attempt to reply
> > to the big picture seen in your etherpad. I was tempted to insert
> > comments on each lines of it, but I'm not sure how this would be
> > received, and probably it's best to attempt to reply more globally.
> >
> > From what I understand, the biggesgt problems you're trying to solve is
> > that managing the global-reqs is really time consuming from the release
> > team point of view, and especially its propagation to individual
> > projects. There's IMO many things that we could do to improve the
> > situation, which would be acceptable from the package maintainers point
> > of view.
> >
> > First of all, from what I could see in the etherpad, I see a lot of
> > release work which I consider not useful for anyone: not for downstream
> > distros, not upstream projects. Mostly, the propagation of the
> > global-requirements.txt to each and every individual Python library or
> > service *for OpenStack maintained libs* could be reviewed. Because 1/
> > distros will always package the highest version available in
> > upper-constraints.txt, and 2/ it doesn't really reflect a reality. As
> > you pointed out, project A may need a new feature from lib X, but
> > project B wont care. I strongly believe that we should leave lower
> > boundaries as a responsibility of individual projects. What important
> > though, is to make sure that the highest version released does work,
> > because that's what we will effectively package.
> >
> > What we can then later on do, at the distribution level, is artificially
> > set the lower bounds of versions to whatever we've just uploaded for a
> > given release of OpenStack. In fact, I've been doing this a lot already.
> > For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was
> > never anything in the between. Therefore, doing a dependency like:
> >
> > Depends: python-eventlet (>= 0.18.3)
> >
> > makes no sense, and I always pushed:
> >
> > Depends: python-eventlet (>= 0.18.4)
> >
> > as this reflects the reality of distros.
> >
> > If we generalize this concept, then I could push the minimum version of
> > all oslo libs into every single package for a given version of OpenStack.
> >
> > What is a lot more annoying though, is for packages which I do not
> > control directly, and which are used by many other non-OpenStack
> > packages inside the distribution. For example, Django, SQLAlchemy or
> > jQuery, to only name a few.
> >
> > I have absolutely no problem upping the lower bounds for all of
> > OpenStack components aggressively. We don't have gate jobs for the lower
> > bounds of our requirements. If we decide that it becomes the norm, I can
> > generalize and push for doing this even more. For example, after pushing
> > the update of an oslo lib B version X, I could push such requirements
> > everywhere, which in fact, would be a good thing (as this would trigger
> > rebuilds and recheck of all unit tests). Though, all of this would
> > benefit from a lot of automation and checks.
> >
> > On your etherpad, you wrote:
> >
> > "During the lead-up to preparing the final releases, one of the tracking
> > tasks we have is to ensure all projects have synced their global
> > requirements updates. This is another area where we could reduce the
> > burden on the release team."
> >
> > Well, don't bother, this doesn't reflect a reality anyway (ie: maybe
> > service X can use an older version of oslo.utils), so that's not really
> > helpful in any way.
> >
> > You also wrote:
> >
> > "Current ranges in global-requirements are large but most projects do
> > not actively test the oldest supported version (or other versions in
> > between) meaning that the requirement might result in broken packages."
> >
> > Yeah, that's truth, I've seen this and reported a few bugs (the last I
> > have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still
> > very useful hints for package maintainers *for 3rd party libs* (as I
> > wrote, it's less important for OpenStack components). We have a few
> > breakage here and there, but they are hopefully fixes.
> >
> > Though having a single version that projects are allowed to test with is
> > super important, so we can check everything can work together. IMO,
> > that's the part which should absolutely not change. Dropping that is
> > like opening a Pandora box. Relying on containers and/or venv will
> > unfortunately, not work, from my stand point.
> >
> > The general rule for a distribution is that the highest version always
> > win, otherwise, it's never maintainable (for security and bug fixes). It
> > should be the case for *any program*, not even just any OpenStack
> > components. There's never a case where it's ok to use something older,
> > just because it feels like less work to do. This type of "laziness"
> > leads to very dangerous outcomes, always.
> >
> > Though I don't see any issue with a project willing to keep backward
> > compatibility with a lower version than what other project do. In fact,
> > that's highly desirable to always try to remain compatible with lower
> > versions as much as possible, *if* it doesn't bring too much burden
> > (think: we still have loads of Python 2.6 stuff like discover, argparse
> > and such that needs to be cleaned-up).
> >
> > For the projects who aren't following the release cycles, it's ok if
> > they bump their lower bounds at the condition that they keep what's in
> > the upper-constraints.txt, so that we can release them together with the
> > rest of OpenStack. For example, it's ok to require the latest oslo.utils
> > 3.8.0 for Ironic 5.1.0.
> >
> > I also see no reason to completely freeze versions to the point were it
> > is impossible fix bugs in the versionnings. I've seen that multiple
> > times. At the distro level, we do address the issue anyway, so why
> > bother? The only thing that counts for us, is to be able to release
> > everything together. A good example would be that Neutron needing SQLA
> > >= 1.0.12: we refuse to fix it, but the issue is still there, and we
> > have to fix it in distros.
> >
> > Last, before I reply to each individual message: the attempt from Robert
> > to test on lower bounds is really something that should be pushed
> > forward. I would help a lot to actually know what the reality is, rather
> > than doing second-guess and artificially set lower bounds for versions
> > which we think may probably work.
> >
> > On 04/18/2016 02:22 PM, Chris Dent wrote:
> > > targeting an
> > > all-in-one install seems only to serve the purposes of all-in-one rpm-
> > > or deb-based installations.
> >
> > Though that's what most everyone consumes. Or are you proposing that we
> > completely stop publishing OpenStack in distros?
> >
> > Remember that in distros, there's only a single version of a library at
> > any given time, at the exception of transitions (yes, in Red Hat it's
> > technically possible to install multiple versions, but the policy
> > strongly advocates against this).
> >
> > Also, all-in-one is what I use in Debian in my CI, to make sure that
> > everyone works together, with whatever is uploaded (see the
> > openstack-deploy package in Debian).
> >
> > > Many (most?) people won't be doing those kinds of installations.
> >
> > Most users are consuming packages from distributions. Also, if you're
> > using containers, probably you will also prefer using these packages to
> > build your containers: that's the most easy, safe and fast way to build
> > your containers.
> >
> > > If all-in-one installations are important to the rpm- and deb- based
> > > distributions then _they_ should be resolving the dependency issues
> > > local to their own infrastructure
> >
> > Who is "they"? Well, approximately 1 person full time for Debian, 1 for
> > Ubuntu if you combine Corey and David (and Debian and Ubuntu
> > dependencies are worked on together in Debian), so that's 2 full time
> > for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is
> > still accurate.
> >
> > So "we" wont handle it, even if "we" care, because we're already
> > understaffed.
> >
> > > (or realizing that it is too painful and start
> > > containerizing or otherwise as well).
> >
> > Distributions don't package containers, we offer packages that you can
> > use to build them. The way you deploy is IMO orthogonal to packaging.
> >
> > > I think making these changes will help to improve and strengthen the
> > > boundaries and contracts between services. If not technically then
> > > at least socially, in the sense that the negotiations that people
> > > make to get things to work are about what actually matters in their
> > > services, not unwinding python dependencies and the like.
> >
> > Yes, it simplifies the life of developers. But it's going to be hell on
> > earth for production if we discover a serious vulnerability in a
> > component, which will be deployed N times, with N versions. Are we then
> > going to provide N patches? The vulnerability management team is also
> > understaffed...
> >
> > > One challenge I've run into the past is when devstack
> > > plugin A has made an assumption about having access to a python
> > > script provided by devstack plugin B, but it's not on $PATH or its
> > > dependencies are not in the site-packages visible to the current
> > > context. The solution here is to use full paths _into_ virtenvs.
> >
> > Please take a step back. Devstack and virtualenv are for development,
> > not for production. That's not, and should not become, our target.
> > Applying the reasoning you have there will *not* work for distributions.
> > Hopefully, that's what Doug is proposing, as he would still like things
> > to work for distros (right?).
> >
> > On 04/18/2016 03:24 PM, Hayes, Graham wrote:
> > > It is also worth noting that according to the OpenStack User Survey
> > > [0] 56% of deployments use "Unmodifed packages from the operating
> > > system".
> >
> > It is my believe (and hope, as this would mean more users) that some
> > OpenStack users don't even know about this survey, and just install
> > what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed
> > packages" is the answer. Maybe some are deploying "slightly modified
> > packages, but mostly what's upstream", which would bring the score even
> > higher.
> >
> > On 04/18/2016 07:33 PM, Doug Hellmann wrote:
> > > Can you truly not imagine any other useful way to package OpenStack
> > > other than individual packages with shared dependencies that would
> > > be acceptable?
> >
> > I could imagine that, but that's not at all what OpenStack is currently.
> > Currently, each and every service is using Oslo, clients, and so on,
> > plus the glue libs like os-brick, Castellan, you name it.
> >
> > Or did I misunderstood your sentence above?
> >
> > On 04/18/2016 07:49 PM, Sean Dague wrote:
> > > A lot of distros specifically have policies against this kind of
> > > bundling as well, because of security issues like this (which was so
> > > very bad) - http://www.zlib.net/advisory-2002-03-11.txt
> >
> > Gentoo is probably a special case (because building from source and so
> > on), but I'd say that *ALL* distro do have such policy in place (at
> > least, Red Hat, Debian, Ubuntu, SuSE...).
> >
> > On 04/18/2016 08:30 PM, Doug Hellmann wrote:
> > > Our upstream
> > > solution can be pretty light-weight, and that leaves room for
> > > downstream folks to make different choices.
> >
> > I'm not sure what choice we'd have, but to face loads of issues and
> > finally give-up packaging OpenStack if we have python module versions
> > that are conflicting. Hopefully, your proposal wont allow that?
> >
> > On 04/18/2016 08:40 PM, Doug Hellmann wrote:
> > > Some folks seem to be conflating "upstream wants to stop worrying
> > > about co-installability" with "upstream wants us to bundle
> > > dependencies". That's not what I'm proposing.
> >
> > Well, if we need multiple versions of the same Python module, what is
> > the solution in production then?
> >
> > > Another alternate that might work is for downstream folks to do
> > > their own dependency management.
> >
> > I fail to understand what you think distros could do to fix the
> > situation of conflicting versions, other than bundling, which isn't
> > acceptable. Could you explain what you mean by "downstream folks to do
> > their own dependency management"?
> >
> > > That's less optimal, and may not
> > > change the outcome for Gentoo or Debian, where the downstream teams
> > > are small (1 person each, I think?).
> >
> > You can also add "Ubuntu" in the list here, as absolutely all OpenStack
> > dependencies are maintained mostly by me, within Debian, and then later
>
> "absolutely all" of OpenStack's dependencies are not maintained by you in
> Debian. A significant number are maintained by the DPMT (Debian Python
> Modules Team). The large majority are maintained by you, but not
> "absolutely all".
>
> > synced to Ubuntu, with a bit of help from Ubuntu folks (but they are
>

And if you grep the changelogs for openstack deps in Debian you'd see a lot
more than "a bit" of contributions from Ubuntu folks.  But that's not the
point of this thread..

> mostly busy with Canonical own products, and I'm the person doing most
> > of the work, still). This is slowly shifting to a more spread workload,
> > but unfortunately, we're not there yet.
> >
> > As for Red Hat / RDO, Haikel told me they are "2 and a half", so
> > basically, the situation there (unless it changed) is approximately the
> > same amount of burnout people.
> >
> > > I started the discussion to solicit ideas, but I would prefer to
> > > avoid proposing what downstream should do because (a) I'm sure
> > > folks want options and (b) I'm sure I'm not the person to identify
> > > those options.
> >
> > Well, I'm doing the packaging, and I'm not sure if we have any option
> > either, if 2 python dependencies are conflicting.
> >
> > On 04/18/2016 09:10 PM, Jeremy Stanley wrote:
> > > On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote:
> > >> Ya, I'd be happy to work more with upstream. I already review the
> > >> stable-reqs updates and watch them for the stable branches I package
> > >> for. Not sure what else is needed.
> > >
> > > Reviewing the master branch openstack/requirements repository
> > > changes (to make sure deps being added are going to be sane things
> > > for someone in your distro to maintain packages of in the long term)
> > > would also make sense.
> >
> > I've done some of that, and used to be reactive on it. But mostly, the
> > way it's been done so far is very good (thanks to a lot of folks working
> > on it), and it's taking too much of my time to review them. So I just
> > pick it up before packaging a (beta-)version of OpenStack.
> >
> > > To be clear, "Moving the burden to packagers" is not the only option
> > > available to us. I've proposed one option for eliminating the issue,
> > > which has some benefits for us upstream but obviously introduces
> > > some other issues we would need to resolve. Another option is for
> > > more people to get involved in managing the dependency list. Some
> > > (most? all?) of those new people may come from distros, and sharing
> > > the effort among them would make it easier than each of them doing
> > > all of the work individually. Sort of like an open source project.
> >
> > There's 2 options I see to solve the "people reviewing the
> > global-requirements.txt changes aren't enough".
> >
> > 1/ talk to my boss and the folks from Canonical (I can't tell for the
> > RPM world) to allocate more staff for packaging (yes, I'm serious here!
> > I can get you in touch...).
> >
> > 2/ get a lot of dependencies out. Yes, let's really do that! We're
> > constantly adding more, I've seen only very few going away. Currently
> > the global-requirements.txt for stable/mitaka is 377 lines long. If you
> > grep out "client", "oslo" and "os-", "openstack", comments, empty lines,
> > and a few others, there's about 260 dependencies left. That's insane at
> > multiple levels. When are we going to see a movement to clean this mess,
> > and define real OpenStack standards?
> >
> > Cheers,
> >
> > Thomas Goirand (zigo)
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> --
> Ian Cordasco
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Corey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160420/87bd122d/attachment.html>