Open Stack

Excerpts from Matthew Thode's message of 2016-04-18 11:22:38 -0700:
> On 04/18/2016 12:33 PM, Doug Hellmann wrote:
> > Excerpts from Matthew Thode's message of 2016-04-18 10:23:37 -0500:
> >> On 04/18/2016 08:24 AM, Hayes, Graham wrote:
> >>> On 18/04/2016 13:51, Sean Dague wrote:
> >>>> On 04/18/2016 08:22 AM, Chris Dent wrote:
> >>>>> On Mon, 18 Apr 2016, Sean Dague wrote:
> >>>>>
> >>>>>> So if you have strong feelings and ideas, why not get them out in email
> >>>>>> now? That will help in the framing of the conversation.
> >>>>>
> >>>>> I won't be at summit and I feel pretty strongly about this topic, so
> >>>>> I'll throw out my comments:
> >>>>>
> >>>>> I agree with the basic premise: In the big tent universe co-
> >>>>> installability is holding us back and is a huge cost in terms of spent
> >>>>> energy. In a world where service isolation is desirable and common
> >>>>> (whether by virtualenv, containers, different hosts, etc) targeting an
> >>>>> all-in-one install seems only to serve the purposes of all-in-one rpm-
> >>>>> or deb-based installations.
> >>>>>
> >>>>> Many (most?) people won't be doing those kinds of installations. If all-in-
> >>>>> one installations are important to the rpm- and deb- based distributions
> >>>>> then _they_ should be resolving the dependency issues local to their own
> >>>>> infrastructure (or realizing that it is too painful and start
> >>>>> containerizing or otherwise as well).
> >>>>>
> >>>>> I think making these changes will help to improve and strengthen the
> >>>>> boundaries and contracts between services. If not technically then
> >>>>> at least socially, in the sense that the negotiations that people
> >>>>> make to get things to work are about what actually matters in their
> >>>>> services, not unwinding python dependencies and the like.
> >>>>>
> >>>>> A lot of the basics of getting this to work are already in place in
> >>>>> devstack. One challenge I've run into the past is when devstack
> >>>>> plugin A has made an assumption about having access to a python
> >>>>> script provided by devstack plugin B, but it's not on $PATH or its
> >>>>> dependencies are not in the site-packages visible to the current
> >>>>> context. The solution here is to use full paths _into_ virtenvs.
> >>>>
> >>>> As Chris said, doing virtualenvs on the Devstack side for services is
> >>>> pretty much there. The team looked at doing this last year, then stopped
> >>>> due to operator feedback.
> >>>>
> >>>> One of the things that gets a little weird (when using devstack for
> >>>> development) is if you actually want to see the impact of library
> >>>> changes on the environment. As you'll need to make sure you loop and
> >>>> install those libraries into every venv where they are used. This
> >>>> forward reference doesn't really exist. So some tooling there will be
> >>>> needed.
> >>>>
> >>>> Middleware that's pushed from one project into another (like Ceilometer
> >>>> -> Swift) is also a funny edge case that I think get funnier here.
> >>>>
> >>>> Those are mostly implementation details, that probably have work
> >>>> arounds, but would need people on them.
> >>>>
> >>>>
> >>>>  From a strategic perspective this would basically make traditional Linux
> >>>> Packaging of OpenStack a lot harder. That might be the right call,
> >>>> because traditional Linux Packaging definitely suffers from the fact
> >>>> that everything on a host needs to be upgraded at the same time. For
> >>>> large installs of OpenStack (especially public cloud cases) traditional
> >>>> packages are definitely less used.
> >>>>
> >>>> However Linux Packaging is how a lot of people get exposed to software.
> >>>> The power of onboarding with apt-get / yum install is a big one.
> >>>>
> >>>> I've been through the ups and downs of both approaches so many times now
> >>>> in my own head, I no longer have a strong preference beyond the fact
> >>>> that we do one approach today, and doing a different one is effort to
> >>>> make the transition.
> >>>>
> >>>>     -Sean
> >>>>
> >>>
> >>> It is also worth noting that according to the OpenStack User Survey [0]
> >>> 56% of deployments use "Unmodifed packages from the operating system".
> >>>
> >>> Granted it was a small sample size (302 responses to that question)
> >>> but it is worth keeping this in mind as we talk about moving the burden
> >>> to packagers.
> >>>
> >>> 0 - 
> >>> https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf (page 
> >>> 36)
> >>>
> >>> __________________________________________________________________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >> To add to this, I'd also note that I as a packager would likely stop
> >> packaging Openstack at whatever release this goes into.  While the
> >> option to package and ship a virtualenv installed to /usr/local or /opt
> >> exists bundling is not something that should be supported given the
> >> issues it can have (update cadence and security issues mainly).
> > 
> > That's a useful data point, but it comes across as a threat and I'm
> > having trouble taking it as a constructive comment.
> > 
> > Can you truly not imagine any other useful way to package OpenStack
> > other than individual packages with shared dependencies that would
> > be acceptable?
> > 
> > Doug
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> As Sean stated, I didn't mean to come across as a threat, but it is the
> most likely outcome of doing this.  Yes, I could technically package a
> venv but then I'd be concerned about security issues that come with
> bundling.  Also, at least specifically to gentoo, while we can bundle
> software and install that it's highly looked down upon.
> 
> Some of this is specific to shared system libraries (c and the like) but
> here's a couple of links.
> 
> https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies
> https://blog.flameeyes.eu/2009/01/bundling-libraries-for-despair-and-insecurity
> 

Neither of these acknowledge the rather successful shift that has been
made by distros to simply ship the latest versions of fast moving
targets. Firefox has been careful not to break users, and as a result,
Ubuntu can ship the latest Firefox no matter what the underlying OS is.
The only real way this can happen is with bundling, but that's a
responsibility that Firefox has taken on.

Likewise, I think OpenStack would have to take on a similarly greater
responsibility in dependency management. If we decide to depend on the
latest whizbang C library or Python library in Nova, we're accepting that
we will likely need to help ship static versions of those and express
which ones are well tested upstream.

What I think people are suggesting, is that we're comfortable with that
responsibility, because we're finding the other one, keeping everything
co-installable with shared libraries, untenable.