Open Stack

Excerpts from Matthew Thode's message of 2015-10-15 17:40:27 -0500:
> On 10/15/2015 05:29 PM, Robert Collins wrote:
> > On 16 October 2015 at 11:21, Matthew Thode <prometheanfire at gentoo.org> wrote:
> >> On 10/15/2015 05:12 PM, Robert Collins wrote:
> >>> On 16 October 2015 at 08:10, Matthew Thode <prometheanfire at gentoo.org> wrote:
> >>>> On 10/15/2015 02:04 PM, Robert Collins wrote:
> >>> ...
> >>>>>> Where are my caps?
> >>>>>
> >>>>> The known good versions of dependencies for liberty are
> >>>>> http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=stable/liberty
> >>>>>
> >>>> That's good, does that represent an upper constraint for the lower
> >>>> constraint imposed by by the package?  Why is this kept separate?
> >>>> Keeping it separate means that it's not trivial to merge them with
> >>>> what's in each package's requirements.
> >>>
> >>> It represents *one* known good combination of packages. We know that
> >>> that combination passed CI, and we then do all our tests with it. To
> >>> change it, we run it past CI and only move onto using the new set when
> >>> its passed CI.
> >>>
> >>> If we merged it to each packages requirements, we'd reintroduce the
> >>> deadlock that caused so much grief only 6 months back - I don't see
> >>> any reason, desire, or tolerance for doing that upstream.
> >>>
> >>> Its kept separate because it requires 2N commits to shift known-good
> >>> caps around for N repos using per-repo rules.
> >>>
> >>> With hundreds of repositories, it takes us hundreds of commits in two
> >>> batches - and a round trip time of 2 hours per batch (check + gate) to
> >>> shift *a single* requirement. With hundreds of dependencies, thats an
> >>> intolerable amount of overhead.
> >>>
> >>
> >> ya, that is annoying, unfortunately packages don't need to know *ONE*
> >> good combination, they need to know them all, or at least the cap.  What
> >> you are basically doing is shifting all of this extra work onto the
> >> packagers, in fact I wouldn't be surprised if they needed to start
> >> vendoring all of openstack in a virtualenv instead of doing actual packages.
> > 
> > Here's an example of the havoc caps cause:
> >  https://review.openstack.org/#/c/234862/
> > 
> > I don't understand the statement about shifting work. Previously the
> > situation was that you had to guess whether a given release of a
> > dependency (both internal and external) worked with e.g. liberty.
> > 
> > Now there is an explicit list of what works.
> > 
> > Isn't that *better* ?
> > 
> Yes, it's good to know what works, does that show multiple versions of
> the same package when multiple are known to work?  If so I can build a
> range from that.  It's not better as it is because I still don't know
> where liberty ends and mitaka begins.  Is there any place I can find that?

In terms of deliverables we're calling part of liberty vs. mitaka,
you can find them in the openstack/releases repository in the YAML
files in the deliverables directory. The results are published to
http://docs.openstack.org/releases/ for human consumption.

Doug

> 
> >> My question remains though, if someone pip installs nova liberty, will
> >> it pull in deps from mitaka?  As it is now, without caps I cannot
> >> reliably package openstack, do you have a solution? I should probably
> >> start removing the liberty packages I did package since upstream seems
> >> so hostile...
> > 
> > If you don't use the constraints file (which is pip consumable and
> > published on the web so it can be used with pip install trivially)
> > then yes, it will install the latest versions of all packages which
> > are presumed to be doing backwards compatible changes. Things that we
> > *know* are going to be broken still get caps - and we accept the cost
> > of the 2-step dance needed to raise them.
> > 
> 
> What about a daily or weekly check without the constraints file so you
> know when something breaks?  This would allow you to at least know when
> you need to place caps and I could consume that.  I'm fine with leaving
> caps off.  If I can consume mitaka deps in liberty then that's great :D
> 
> > The big question here is, I think, 'should we assume OpenStack
> > originated dependencies are better or worse than third party
> > dependencies?' Third party dependencies we presume are backwards
> > compatible (and will thus only break by mistake, which constraints
> > guards against) unless we have reason not to - and we have open ended
> > deps on them. This is where the spec about clients libraries is aimed.
> > 
> > It makes me very sad to know that you consider what we've done as
> > hostile. We did it for a bunch of good reasons:
> > 
> >  - safer release process
> >  - smoother updates for [apt, rpm at least] redistributors
> >  - faster turnaround of dependency usage in CI
> >  - step on the path to testing lower bounds
> >  - step on the path to alignment with upstream packaging tooling
> > 
> > -Rob
> > 
> > 
> It's a step backwards from my perspective, before I had a clear
> delineation where support of something stops.  Now I don't know when it
> stops and any given update could break the system.  I'm not sure how it
> could be smoother for other systems.
> 
> So, does this mean that I can just leave the packages uncapped and know
> that they will work?  Are there tests being run for this scenario?
>