[openstack-dev] [all][pbr] splitting our deployment vs install dependencies
mordred at inaugust.com
Sun Apr 12 23:12:38 UTC 2015
On 04/12/2015 06:43 PM, Robert Collins wrote:
> Right now we do something that upstream pip considers wrong: we make
> our requirements.txt be our install_requires.
> Upstream there are two separate concepts.
> install_requirements, which are meant to document what *must* be
> installed to import the package, and should encode any mandatory
> version constraints while being as loose as otherwise possible. E.g.
> if package A depends on package B version 1.5 or above, it should say
> B>=1.5 in A's install_requires. They should not specify maximum
> versions except when that is known to be a problem: they shouldn't
> borrow trouble.
> deploy requirements - requirements.txt - which are meant to be *local
> to a deployment*, and are commonly expected to specify very narrow (or
> even exact fit) versions.
tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.
To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
Nor, btw, is it the right split. It optimizes for the wrong thing.
rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
_application_ should ship with a Cargo.lock and a _library_ should not.
Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.
> What pbr, which nearly if not all OpenStack projects use, does, is to
> map the contents of requirements.txt into install_requires. And then
> we use the same requirements.txt in our CI to control whats deployed
> in our test environment[*]. and there we often have tight constraints
> like seen here -
That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
> I'd like to align our patterns with those of upstream, so that we're
> not fighting our tooling so much.
Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.
> Concretely, I think we need to:
> - teach pbr to read in install_requires from setup.cfg, not requirements.txt
> - when there are requirements in setup.cfg, stop reading requirements.txt
> - separate out the global intall_requirements from the global CI
> requirements, and update our syncing code to be aware of this
> Then, setup.cfg contains more open requirements suitable for being on
> PyPI, requirements.txt is the local CI set we know works - and can be
> much more restrictive as needed.
> Thoughts? If there's broad apathy-or-agreement I can turn this into a
> spec for fine coverage of ramifications and corner cases.
I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.
I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that _still_
isn't going to be "how people do it"
I'm concerned that "how people do it" is a myth not worth chasing.
I'm not _opposed_ to making this richer and more useful for people. I
just don't know what's broken currently for us.
More information about the OpenStack-dev