Open Stack

On 04/15/2015 06:44 PM, Robert Collins wrote:
> On 16 April 2015 at 01:50, Sean Dague <sean at dague.net> wrote:
>> On 04/12/2015 06:43 PM, Robert Collins wrote:
> 
>>> Thoughts? If there's broad apathy-or-agreement I can turn this into a
>>> spec for fine coverage of ramifications and corner cases.
>>
>> I'm definitely happy someone else is diving in on here, just beware the
>> dragons, there are many.
>>
>> I think some of the key problems are the following (lets call these the
>> requirements requirements):
> 
> :) There's definitely enough meat here we're going to want a spec to
> review in one place the conclusions and inputs.
> 
>> == We would like to be able to install multiple projects into a single
>> devstack instance, and have all services work.
> 
> -- why do we want this? [for completeness - no objecting]. I think we
> want this because its easier for folk mucking around to not have to
> remember which venv etc; because we have projects like neutron and
> nova that install bits into each others processes via common
> libraries; because deployers have asked us to be sure that we can
> co-install everything.

I think the completeness statement here is as follows:

1. For OpenStack to scale to the small end, we need to be able to
overlap services, otherwise you are telling people they basically have
to start with a full rack of hardware to get 1 worker.

2. The Linux Distributors (who are a big part of our community) install
everything at a system level. Python has no facility for having multiple
versions of libraries installed at the system level, largely because
virtualenvs were created (which solve the non system application level
problem really well).

3. The alternative of putting a venv inside of every service violates
the principle of single location security update.

Note: there is an aside about why we might *not* want to do this....

That being said, if you deploy at a system level your upgrade unit is
now 1 full box, instead of 1 service, because you can't do library
isolation between old and new. A worker might have neutron, cinder, and
nova agents. Only the Nova agents support rolling upgrade (cinder is
working hard on this, I don't think neutron has visited this yet). So
real rolling upgrade is sacrificed on this alter of install everything
at a system level.

> 
>> This is hard because:
>>
>> 1. these are multiple projects so pip can't resolve all requirements at
>> once to get to a solved state (also, optional dependencies in particular
>> configs mean these can be installed later)
> 
> I don't understand your first clause here. Pip certainly can resolve
> all requirements at once: for instance, 'pip install path_to_nova
> path_to_swift path_to_neutron' would resolve all the requirements for
> all three at once. We're not doing that today, but its not a pip
> limitation. Today https://github.com/pypa/pip/issues/2687 will rear
> its head, but that may be quite shallow.
> 
> As far as optional deps go - we need to get those into extra
> requirements, then pip can examine that for us. Enabling that is on my
> stack that I'm rolling up at the moment.
> 
>> 2. pip's solver ignores setup_requires - https://github.com/pypa/pip/issues
>> /2612#issuecomment-91114298 which means we can get inconsistent results
> 
> Ish. The actual issue we ran into was
> https://bitbucket.org/pypa/setuptools/issue/141/setup_requires-feature-does-not-handle
> . We can tackle that directly and then require a newer setuptools to
> solve this - it doesn't need any larger context.
> 
>> 3. doing this iteratively in projects can cause the following to happen
>>
>> A requires B>1.0,<2.0
>> C requires B>1.2
>>
>> pip install C can make the pip install A requirements invalid later.
> 
> https://github.com/pypa/pip/issues/2687 again. I suspect this is a ~10
> line patch - read all the package metadata present on the system, and
> union its deps in inside PackageSet.add_requirement.
> 
>> This can end up in a failure of a service to start (if pkg_resources is
>> actually checking things), or very subtle bugs later.
>>
>> Today global-requirements attempts to address this by continuously
>> narrowing the requirements definitions for everything we have under our
>> control so that pip is living in a rubber room and can only get an
>> answer we know works.
> 
>> == However.... this has exposed an additional issue, libraries not
>> released at release time
>>
>> Way more things are getting g-r syncs than top level projects.
>> Synchronizing requirements for things that all release at the same time
>> makes a lot of sense. However we're synchronizing requirements into
>> libraries that release at different cadence. This has required all
>> libraries to also have stable/ branches, for requirements matching.
>>
>> In an ideal world libraries would have very broad requirements, which
>> would not have caps in them. non library projects would have narrower
>> requirements that we know work.
> 
> I mostly agree. That is I think the heart of the issue I started this
> thread about.
> For libraries I trivially agree.
> For non-library projects, I think we still need to be Known-Not-Bad,
> vs Known-Good, but for CI our overrides can resolve that into
> Known-Good - and we can publish this in some well known,
> automation-friendly way.
> 
> Concretely, devstack should be doing one pip install run, and in
> stable branches that needs to look something like:
> 
> $ pip install -r known-good-list $path_to_nova $path_to_neutron ....

Also remember we need to -e install a bunch of directories, not sure if
that makes things easier or harder.

So, one of the problems with that is often what we need is buried pretty
deep like -
https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L39

If devstack needs to uplift everything into an install the world phase,
I think we're basically on the verge of creating OpenStack Package
Manager so that we can specify this all declaratively (after processing
a bunch of conditional feature logic). Which, is a thing someone could
do (not it), but if we really get there we're probably just better off
thinking about reviving building debs so that we can have the package
manager globally keep track of all the requirements across multiple
invocations.

You are right, https://github.com/pypa/pip/issues/988 "Pip needs a
dependency resolver" ends up being a problem for us regardless. And
based on reading through it again tonight it seems pretty clear that a
number of other complicated system level python programs are running
head long into the same issues.

	-Sean

-- 
Sean Dague
http://dague.net