Open Stack

On 10/15/2015 05:29 PM, Robert Collins wrote:
> On 16 October 2015 at 11:21, Matthew Thode <prometheanfire at gentoo.org> wrote:
>> On 10/15/2015 05:12 PM, Robert Collins wrote:
>>> On 16 October 2015 at 08:10, Matthew Thode <prometheanfire at gentoo.org> wrote:
>>>> On 10/15/2015 02:04 PM, Robert Collins wrote:
>>> ...
>>>>>> Where are my caps?
>>>>>
>>>>> The known good versions of dependencies for liberty are
>>>>> http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=stable/liberty
>>>>>
>>>> That's good, does that represent an upper constraint for the lower
>>>> constraint imposed by by the package?  Why is this kept separate?
>>>> Keeping it separate means that it's not trivial to merge them with
>>>> what's in each package's requirements.
>>>
>>> It represents *one* known good combination of packages. We know that
>>> that combination passed CI, and we then do all our tests with it. To
>>> change it, we run it past CI and only move onto using the new set when
>>> its passed CI.
>>>
>>> If we merged it to each packages requirements, we'd reintroduce the
>>> deadlock that caused so much grief only 6 months back - I don't see
>>> any reason, desire, or tolerance for doing that upstream.
>>>
>>> Its kept separate because it requires 2N commits to shift known-good
>>> caps around for N repos using per-repo rules.
>>>
>>> With hundreds of repositories, it takes us hundreds of commits in two
>>> batches - and a round trip time of 2 hours per batch (check + gate) to
>>> shift *a single* requirement. With hundreds of dependencies, thats an
>>> intolerable amount of overhead.
>>>
>>
>> ya, that is annoying, unfortunately packages don't need to know *ONE*
>> good combination, they need to know them all, or at least the cap.  What
>> you are basically doing is shifting all of this extra work onto the
>> packagers, in fact I wouldn't be surprised if they needed to start
>> vendoring all of openstack in a virtualenv instead of doing actual packages.
> 
> Here's an example of the havoc caps cause:
>  https://review.openstack.org/#/c/234862/
> 
> I don't understand the statement about shifting work. Previously the
> situation was that you had to guess whether a given release of a
> dependency (both internal and external) worked with e.g. liberty.
> 
> Now there is an explicit list of what works.
> 
> Isn't that *better* ?
> 
Yes, it's good to know what works, does that show multiple versions of
the same package when multiple are known to work?  If so I can build a
range from that.  It's not better as it is because I still don't know
where liberty ends and mitaka begins.  Is there any place I can find that?

>> My question remains though, if someone pip installs nova liberty, will
>> it pull in deps from mitaka?  As it is now, without caps I cannot
>> reliably package openstack, do you have a solution? I should probably
>> start removing the liberty packages I did package since upstream seems
>> so hostile...
> 
> If you don't use the constraints file (which is pip consumable and
> published on the web so it can be used with pip install trivially)
> then yes, it will install the latest versions of all packages which
> are presumed to be doing backwards compatible changes. Things that we
> *know* are going to be broken still get caps - and we accept the cost
> of the 2-step dance needed to raise them.
> 

What about a daily or weekly check without the constraints file so you
know when something breaks?  This would allow you to at least know when
you need to place caps and I could consume that.  I'm fine with leaving
caps off.  If I can consume mitaka deps in liberty then that's great :D

> The big question here is, I think, 'should we assume OpenStack
> originated dependencies are better or worse than third party
> dependencies?' Third party dependencies we presume are backwards
> compatible (and will thus only break by mistake, which constraints
> guards against) unless we have reason not to - and we have open ended
> deps on them. This is where the spec about clients libraries is aimed.
> 
> It makes me very sad to know that you consider what we've done as
> hostile. We did it for a bunch of good reasons:
> 
>  - safer release process
>  - smoother updates for [apt, rpm at least] redistributors
>  - faster turnaround of dependency usage in CI
>  - step on the path to testing lower bounds
>  - step on the path to alignment with upstream packaging tooling
> 
> -Rob
> 
> 
It's a step backwards from my perspective, before I had a clear
delineation where support of something stops.  Now I don't know when it
stops and any given update could break the system.  I'm not sure how it
could be smoother for other systems.

So, does this mean that I can just leave the packages uncapped and know
that they will work?  Are there tests being run for this scenario?

-- 
Matthew Thode (prometheanfire)