[openstack-dev] [all][pbr] splitting our deployment vs install dependencies

James Polley jp at jamezpolley.com
Mon Apr 13 00:01:44 UTC 2015

On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor <mordred at inaugust.com> wrote:

> On 04/12/2015 06:43 PM, Robert Collins wrote:
> > Right now we do something that upstream pip considers wrong: we make
> > our requirements.txt be our install_requires.
> >
> > Upstream there are two separate concepts.
> >
> > install_requirements, which are meant to document what *must* be
> > installed to import the package, and should encode any mandatory
> > version constraints while being as loose as otherwise possible. E.g.
> > if package A depends on package B version 1.5 or above, it should say
> > B>=1.5 in A's install_requires. They should not specify maximum
> > versions except when that is known to be a problem: they shouldn't
> > borrow trouble.
> >
> > deploy requirements - requirements.txt - which are meant to be *local
> > to a deployment*, and are commonly expected to specify very narrow (or
> > even exact fit) versions.

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the

> tl;dr - I'm mostly in support of what you're saying - but I'm going to
> bludgeon it some.
> To be either fair or unfair, depending on how you look at it - some
> people upstream consider those two to be a pattern, but it is not
> encoded anywhere except in hidden lore that is shared between secret
> people. Upstream's tools have bumpkiss for support for this, and if we
> hadn't drawn a line in the sand encoding SOME behavior there would still
> be nothing.
> Nor, btw, is it the right split. It optimizes for the wrong thing.
> rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
> understood by the tooling in a manner similar to what you have
> described, and it is not just understood but DOCUMENTED that an
> _application_ should ship with a Cargo.lock and a _library_ should not.

This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in

> Without the library/application distinction, the effort in
> differentiating is misplaced, I believe - because it's libraries that
> need flexible depends - and applications where the specific set of
> depends that were tested in CI become important to consumers.
> > What pbr, which nearly if not all OpenStack projects use, does, is to
> > map the contents of requirements.txt into install_requires. And then
> > we use the same requirements.txt in our CI to control whats deployed
> > in our test environment[*]. and there we often have tight constraints
> > like seen here -
> >
> http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
> That is, btw, because that's what the overwhelming majority of consumers
> assume those files mean. I take "overwhelming majority" from the days
> when we had files but did not process them automatically and everyone
> was confused.
> > I'd like to align our patterns with those of upstream, so that we're
> > not fighting our tooling so much.
> Ok. I mean, they don't have a better answer, but if it makes "python"
> hate us less, sweet.
> > Concretely, I think we need to:
> >  - teach pbr to read in install_requires from setup.cfg, not
> requirements.txt
> >  - when there are requirements in setup.cfg, stop reading
> requirements.txt
> >  - separate out the global intall_requirements from the global CI
> > requirements, and update our syncing code to be aware of this
> >
> > Then, setup.cfg contains more open requirements suitable for being on
> > PyPI, requirements.txt is the local CI set we know works - and can be
> > much more restrictive as needed.
> >
> > Thoughts? If there's broad apathy-or-agreement I can turn this into a
> > spec for fine coverage of ramifications and corner cases.
> I'm concerned that it adds a layer of difference that is confusing to
> people for the sole benefit of pleasing someone else's pedantic worldview.
> I'm also concerned that dstufft is actively wanting to move towards a
> world where the build tooling is not needed or used as part of the
> install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
> with a pattern that isn't very robust and isn't supported by tooling
> directly and that we're going to need to change understood usage
> patterns across a large developer based to chase something that _still_
> isn't going to be "how people do it"
> I'm concerned that "how people do it" is a myth not worth chasing.
> I'm not _opposed_ to making this richer and more useful for people. I
> just don't know what's broken currently for us.

To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.

> Monty
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/4e267c09/attachment.html>

More information about the OpenStack-dev mailing list