[openstack-dev] [all][pbr] splitting our deployment vs install dependencies

Robert Collins robertc at robertcollins.net
Mon Apr 13 22:04:50 UTC 2015

On 13 April 2015 at 22:04, Thierry Carrez <thierry at openstack.org> wrote:
>> This observation led to yet more IRC discussion and eventually
>> https://etherpad.openstack.org/p/stable-omg-deps
>> In short, the proposal is that we:
>>  - stop trying to use install_requires to reproduce exactly what
>> works, and instead use it to communicate known constraints (> X, Y is
>> broken etc).
>>  - use a requirements.txt file we create *during* CI to capture
>> exactly what worked, and also capture the dpkg and rpm versions of
>> packages that were present when it worked, and so on. So we'll build a
>> git tree where its history is an audit trail of exactly what worked
>> for everything that passed CI, formatted to make it really really easy
>> for other people to consume.
> I totally agree that we need to stop trying to provide two different
> sets of dependency information (known good deps, known bad deps) using
> the same dataset.
> If I understand you correctly, today we provide a requirements.txt and
> generate an install_requires from it, and in the new world order we
> would provide a install_requires with "known-bad" info in it and
> generate a "known-good" requirements.txt (during CI) from it.

Yes, with two clarifying points: the known-good has to live in a
different repo from the project, because we only discover that during
CI, after the commits have been made. Secondly, the install_requires
will be delivered via setup.cfg in the project tree.

> Questions:
> How would global-requirements evolve in that picture ? Would we have
> some "global-install-requires" thing to replace it ?

I think global-requirements today is (by necessity) mostly known-bad,
and doesn't need to change much. It needs to learn how to reference
setup.cfg metadata as well/rather than {test-}requirements{-pyNN}.txt.
There's a separate discussion we had a few weeks back about
consolidating the non-install-requires we have into setup.cfg with
appropriate tags, which we'll want to do at the same time.

> Distro packagers today rely on requirements.txt (and global
> -requirements) to determine what version of libraries they need to
> package. Would they just rely on install_requires instead ? Where is
> that information provided ? setup.cfg ?

Yes. Project + global-requirements is a good combination. They might
want to reference the known-good exact lists as an additional data

> How does this proposal affect stable branches ? In order to keep the
> breakage there under control, we now have stable branches for all the
> OpenStack libraries and cap accordingly[1]. We planned to cap all other
> libraries to "the version that was there when the stable branch was
> cut".  Where would we do those cappings in the new world order ? In
> install_requires ? Or should we not do that anymore ?
> [1]
> http://specs.openstack.org/openstack/openstack-specs/specs/library-stable-branches.html

I don't think there's a hard and fast answer here. Whats proposed
there should work fine.

On the one hand, semver tells us when *a* backwards compat break
happens, but it doesn't tell us if *that break affects user X*. For
instance, the general migration pattern we expect is:
 - introduce new API  V=2.3.0
 - migrate all our users V~=2.3
 - deprecate old API     V~=2.3
 - gc deprecated code at some future date V>=3.0

In fact, I'd say we're hoping to never have a supported release broken
by that process... so capping just creates artificial version
conflicts which we have to resolve by issuing updates to say that
actually the new major version is still compatible with this new

OTOH there will eventually be releases of our libraries that do break
prior releases of our servers/clients - and when that happens capped
requirements will actually be useful, but only to people running
unsupported releases :).

OTGH if we do deliberately break supported releases in our libraries,
then the capping process is absolutely essential.

Personally, I'd be more worried about the versions of our dependencies
that *aren't* coordinated with our projects, because if they aren't
capped, (and they're doing semver) we're less likely to find out the
easy way (in advance :)) about issues.

But that then swings back around to known good vs known bad. One way
of looking at that is that safe capping requires several items of
- what version to use with ~= - I'm not sure that using the exact
version we got is correct. e.g. with semver, if 1.2.3 is known-good,
we should use ~=1.2 (e.g. >=1.2, ==1.*), but with date based its
harder to predict what will indicate a breaking version :). And of
course for non-semver, 1.2.3 doesn't tell us whether 1.3 will be
breaking, or even 1.2.4.
- a known good version to base our cap on

If we generated the first item and stored it somewhere, then when we
generate known-good == lists from CI, we could also generate a
known-good capped list, (e.g. transforming 1.2.3 to ~=1.2 for semver
projects). We could in principle add that to our tarball releases of
projects, even though we can't sensibly put it in git.

tl;dr - I dunno :)


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

More information about the OpenStack-dev mailing list