Open Stack

On 16 October 2015 at 11:21, Matthew Thode <prometheanfire at gentoo.org> wrote:
> On 10/15/2015 05:12 PM, Robert Collins wrote:
>> On 16 October 2015 at 08:10, Matthew Thode <prometheanfire at gentoo.org> wrote:
>>> On 10/15/2015 02:04 PM, Robert Collins wrote:
>> ...
>>>>> Where are my caps?
>>>>
>>>> The known good versions of dependencies for liberty are
>>>> http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=stable/liberty
>>>>
>>> That's good, does that represent an upper constraint for the lower
>>> constraint imposed by by the package?  Why is this kept separate?
>>> Keeping it separate means that it's not trivial to merge them with
>>> what's in each package's requirements.
>>
>> It represents *one* known good combination of packages. We know that
>> that combination passed CI, and we then do all our tests with it. To
>> change it, we run it past CI and only move onto using the new set when
>> its passed CI.
>>
>> If we merged it to each packages requirements, we'd reintroduce the
>> deadlock that caused so much grief only 6 months back - I don't see
>> any reason, desire, or tolerance for doing that upstream.
>>
>> Its kept separate because it requires 2N commits to shift known-good
>> caps around for N repos using per-repo rules.
>>
>> With hundreds of repositories, it takes us hundreds of commits in two
>> batches - and a round trip time of 2 hours per batch (check + gate) to
>> shift *a single* requirement. With hundreds of dependencies, thats an
>> intolerable amount of overhead.
>>
>
> ya, that is annoying, unfortunately packages don't need to know *ONE*
> good combination, they need to know them all, or at least the cap.  What
> you are basically doing is shifting all of this extra work onto the
> packagers, in fact I wouldn't be surprised if they needed to start
> vendoring all of openstack in a virtualenv instead of doing actual packages.

Here's an example of the havoc caps cause:
 https://review.openstack.org/#/c/234862/

I don't understand the statement about shifting work. Previously the
situation was that you had to guess whether a given release of a
dependency (both internal and external) worked with e.g. liberty.

Now there is an explicit list of what works.

Isn't that *better* ?

> My question remains though, if someone pip installs nova liberty, will
> it pull in deps from mitaka?  As it is now, without caps I cannot
> reliably package openstack, do you have a solution? I should probably
> start removing the liberty packages I did package since upstream seems
> so hostile...

If you don't use the constraints file (which is pip consumable and
published on the web so it can be used with pip install trivially)
then yes, it will install the latest versions of all packages which
are presumed to be doing backwards compatible changes. Things that we
*know* are going to be broken still get caps - and we accept the cost
of the 2-step dance needed to raise them.

The big question here is, I think, 'should we assume OpenStack
originated dependencies are better or worse than third party
dependencies?' Third party dependencies we presume are backwards
compatible (and will thus only break by mistake, which constraints
guards against) unless we have reason not to - and we have open ended
deps on them. This is where the spec about clients libraries is aimed.

It makes me very sad to know that you consider what we've done as
hostile. We did it for a bunch of good reasons:

 - safer release process
 - smoother updates for [apt, rpm at least] redistributors
 - faster turnaround of dependency usage in CI
 - step on the path to testing lower bounds
 - step on the path to alignment with upstream packaging tooling

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud