Open Stack

On 04/12/2015 06:43 PM, Robert Collins wrote:
> Right now we do something that upstream pip considers wrong: we make
> our requirements.txt be our install_requires.
> 
> Upstream there are two separate concepts.
> 
> install_requirements, which are meant to document what *must* be
> installed to import the package, and should encode any mandatory
> version constraints while being as loose as otherwise possible. E.g.
> if package A depends on package B version 1.5 or above, it should say
> B>=1.5 in A's install_requires. They should not specify maximum
> versions except when that is known to be a problem: they shouldn't
> borrow trouble.
> 
> deploy requirements - requirements.txt - which are meant to be *local
> to a deployment*, and are commonly expected to specify very narrow (or
> even exact fit) versions.
> 
> What pbr, which nearly if not all OpenStack projects use, does, is to
> map the contents of requirements.txt into install_requires. And then
> we use the same requirements.txt in our CI to control whats deployed
> in our test environment[*]. and there we often have tight constraints
> like seen here -
> http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
> 
> I'd like to align our patterns with those of upstream, so that we're
> not fighting our tooling so much.
> 
> Concretely, I think we need to:
>  - teach pbr to read in install_requires from setup.cfg, not requirements.txt
>  - when there are requirements in setup.cfg, stop reading requirements.txt
>  - separate out the global intall_requirements from the global CI
> requirements, and update our syncing code to be aware of this
> 
> Then, setup.cfg contains more open requirements suitable for being on
> PyPI, requirements.txt is the local CI set we know works - and can be
> much more restrictive as needed.
> 
> Thoughts? If there's broad apathy-or-agreement I can turn this into a
> spec for fine coverage of ramifications and corner cases.

I'm definitely happy someone else is diving in on here, just beware the
dragons, there are many.

I think some of the key problems are the following (lets call these the
requirements requirements):

== We would like to be able to install multiple projects into a single
devstack instance, and have all services work.

This is hard because:

1. these are multiple projects so pip can't resolve all requirements at
once to get to a solved state (also, optional dependencies in particular
configs mean these can be installed later)

2. pip's solver ignores setup_requires - https://github.com/pypa/pip/issues
/2612#issuecomment-91114298 which means we can get inconsistent results

3. doing this iteratively in projects can cause the following to happen

A requires B>1.0,<2.0
C requires B>1.2

pip install C can make the pip install A requirements invalid later.
This can end up in a failure of a service to start (if pkg_resources is
actually checking things), or very subtle bugs later.

Today global-requirements attempts to address this by continuously
narrowing the requirements definitions for everything we have under our
control so that pip is living in a rubber room and can only get an
answer we know works.



== However.... this has exposed an additional issue, libraries not
released at release time

Way more things are getting g-r syncs than top level projects.
Synchronizing requirements for things that all release at the same time
makes a lot of sense. However we're synchronizing requirements into
libraries that release at different cadence. This has required all
libraries to also have stable/ branches, for requirements matching.

In an ideal world libraries would have very broad requirements, which
would not have caps in them. non library projects would have narrower
requirements that we know work.


== End game?

*If* pip install took into account the requirements of everything
already installed like apt or yum does, and resolve accordingly
(including saying that's not possible unless you uninstall or upgrade
X), we'd be able to pip install and get a working answer at the end. Maybe?

Honestly, there are so many fixes on fixes here to our system, I'm not
sure even this would fix it.



-- 
Sean Dague
http://dague.net