Open Stack

I think what we are trying to do is two separate things.

One is to define the dependencies that packagers use.  This would likely
be minimum versions with caps that are known to fail (not assumed).

The second is to define a set of verifiably known working deps.  This
would likely need an update mechanism probably so it doesn't stagnate.

What could then be done is to merge the two sources by default or
packagers can just removed the auto-capped file.  Separating it out
allows us to keep track of known bad versions as well.  If I see a cap
on a lib at this point I assume it was a protectionist measure, not
because it was actually a bug or something.

As it is now, it is almost impossible to package 2014.2.3 because at
least my distro has removed a bunch of old libraries that were not
needed because the caps not there.

For kilo packaging will likely be fine because we can lock the deps from
the start so versions we need are not removed.  Our distro at least
allows the package manager to choose which version of a package to
install to meet the requirements.

On 04/13/2015 05:04 PM, Robert Collins wrote:
> On 13 April 2015 at 22:04, Thierry Carrez <thierry at openstack.org> wrote:
>>> This observation led to yet more IRC discussion and eventually
>>> https://etherpad.openstack.org/p/stable-omg-deps
>>>
>>> In short, the proposal is that we:
>>>  - stop trying to use install_requires to reproduce exactly what
>>> works, and instead use it to communicate known constraints (> X, Y is
>>> broken etc).
>>>  - use a requirements.txt file we create *during* CI to capture
>>> exactly what worked, and also capture the dpkg and rpm versions of
>>> packages that were present when it worked, and so on. So we'll build a
>>> git tree where its history is an audit trail of exactly what worked
>>> for everything that passed CI, formatted to make it really really easy
>>> for other people to consume.
>>
>> I totally agree that we need to stop trying to provide two different
>> sets of dependency information (known good deps, known bad deps) using
>> the same dataset.
>>
>> If I understand you correctly, today we provide a requirements.txt and
>> generate an install_requires from it, and in the new world order we
>> would provide a install_requires with "known-bad" info in it and
>> generate a "known-good" requirements.txt (during CI) from it.
> 
> Yes, with two clarifying points: the known-good has to live in a
> different repo from the project, because we only discover that during
> CI, after the commits have been made. Secondly, the install_requires
> will be delivered via setup.cfg in the project tree.
> 
>> Questions:
>>
>> How would global-requirements evolve in that picture ? Would we have
>> some "global-install-requires" thing to replace it ?
> 
> I think global-requirements today is (by necessity) mostly known-bad,
> and doesn't need to change much. It needs to learn how to reference
> setup.cfg metadata as well/rather than {test-}requirements{-pyNN}.txt.
> There's a separate discussion we had a few weeks back about
> consolidating the non-install-requires we have into setup.cfg with
> appropriate tags, which we'll want to do at the same time.
> 
>> Distro packagers today rely on requirements.txt (and global
>> -requirements) to determine what version of libraries they need to
>> package. Would they just rely on install_requires instead ? Where is
>> that information provided ? setup.cfg ?
> 
> Yes. Project + global-requirements is a good combination. They might
> want to reference the known-good exact lists as an additional data
> source.
> 
>> How does this proposal affect stable branches ? In order to keep the
>> breakage there under control, we now have stable branches for all the
>> OpenStack libraries and cap accordingly[1]. We planned to cap all other
>> libraries to "the version that was there when the stable branch was
>> cut".  Where would we do those cappings in the new world order ? In
>> install_requires ? Or should we not do that anymore ?
>>
>> [1]
>> http://specs.openstack.org/openstack/openstack-specs/specs/library-stable-branches.html
> 
> I don't think there's a hard and fast answer here. Whats proposed
> there should work fine.
> 
> On the one hand, semver tells us when *a* backwards compat break
> happens, but it doesn't tell us if *that break affects user X*. For
> instance, the general migration pattern we expect is:
>  - introduce new API  V=2.3.0
>  - migrate all our users V~=2.3
>  - deprecate old API     V~=2.3
>  - gc deprecated code at some future date V>=3.0
> 
> In fact, I'd say we're hoping to never have a supported release broken
> by that process... so capping just creates artificial version
> conflicts which we have to resolve by issuing updates to say that
> actually the new major version is still compatible with this new
> release...
> 
> OTOH there will eventually be releases of our libraries that do break
> prior releases of our servers/clients - and when that happens capped
> requirements will actually be useful, but only to people running
> unsupported releases :).
> 
> OTGH if we do deliberately break supported releases in our libraries,
> then the capping process is absolutely essential.
> 
> Personally, I'd be more worried about the versions of our dependencies
> that *aren't* coordinated with our projects, because if they aren't
> capped, (and they're doing semver) we're less likely to find out the
> easy way (in advance :)) about issues.
> 
> But that then swings back around to known good vs known bad. One way
> of looking at that is that safe capping requires several items of
> data:
> - what version to use with ~= - I'm not sure that using the exact
> version we got is correct. e.g. with semver, if 1.2.3 is known-good,
> we should use ~=1.2 (e.g. >=1.2, ==1.*), but with date based its
> harder to predict what will indicate a breaking version :). And of
> course for non-semver, 1.2.3 doesn't tell us whether 1.3 will be
> breaking, or even 1.2.4.
> - a known good version to base our cap on
> 
> If we generated the first item and stored it somewhere, then when we
> generate known-good == lists from CI, we could also generate a
> known-good capped list, (e.g. transforming 1.2.3 to ~=1.2 for semver
> projects). We could in principle add that to our tarball releases of
> projects, even though we can't sensibly put it in git.
> 
> tl;dr - I dunno :)
> 
> -Rob
> 
> 


-- 
-- Matthew Thode (prometheanfire)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/48ecea67/attachment.pgp>