Open Stack

> -----Original Message-----
> From: Clark Boylan [mailto:cboylan at sapwetik.org]
> Sent: 03 September 2014 21:57
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
> 
> 
> 
> On Wed, Sep 3, 2014, at 01:06 PM, Kuvaja, Erno wrote:
> > > -----Original Message-----
> > > From: Clark Boylan [mailto:cboylan at sapwetik.org]
> > > Sent: 03 September 2014 20:10
> > > To: openstack-dev at lists.openstack.org
> > > Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort
> > > requirements.txt
> > >
> > >
> > >
> > > On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote:
> > > > > -----Original Message-----
> > > > > From: Sean Dague [mailto:sean at dague.net]
> > > > > Sent: 03 September 2014 13:37
> > > > > To: OpenStack Development Mailing List (not for usage questions)
> > > > > Subject: [openstack-dev] [all] [glance] do NOT ever sort
> > > > > requirements.txt
> > > > >
> > > > > I'm not sure why people keep showing up with "sort requirements"
> > > > > patches like - https://review.openstack.org/#/c/76817/6,
> > > > > however, they
> > > do.
> > > > >
> > > > > All of these need to be -2ed with predjudice.
> > > > >
> > > > > requirements.txt is not a declarative interface. The order is
> > > > > important as pip processes it in the order it is. Changing the
> > > > > order has impacts on the overall integration which can cause wedges
> later.
> > > > >
> > > > > So please stop.
> > > > >
> > > > > 	-Sean
> > > > >
> > > > > --
> > > > > Sean Dague
> > > > > http://dague.net
> > > > >
> > > >
> > > > Hi Sean & all,
> > > >
> > > > Could you please open this up a little bit? What are we afraid
> > > > breaking regarding the order of these requirements? I tried to go
> > > > through pip documentation but I could not find reason of specific
> > > > order of the lines, references to keep the order there was 'though.
> > > >
> > > > I'm now assuming one thing here as I do not know if that's the case.
> > > > None of the packages enables/disables functionality depending of
> > > > what has been installed on the system before, but they have their
> > > > own dependencies to provide those. Based on this assumption I can
> > > > think of only one scenario causing us issues. That is us abusing
> > > > the example in point 2 of
> > > > https://pip.pypa.io/en/latest/user_guide.html#requirements-files
> > > > meaning; we install package X depending on package Y>=1.0,<2.0
> > > > before installing package Z depending on Y>=1.0 to ensure that
> > > > package Y<2.0 without pinning package Y in our requirements.txt. I
> > > > certainly hope that this is not the case as depending 3rd party
> > > > vendor providing us specific
> > > version of dependency package would be extremely stupid.
> > > >
> > > > Other than that I really don't know how the order could cause us
> > > > issues, but I would be really happy to learn something new today
> > > > if that is the case or if my assumption went wrong.
> > > >
> > > > Best Regards,
> > > > Erno (jokke_) Kuvaja
> > > >
> > > > > _______________________________________________
> > > > > OpenStack-dev mailing list
> > > > > OpenStack-dev at lists.openstack.org
> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
> > > > > v
> > > >
> > > > _______________________________________________
> > > > OpenStack-dev mailing list
> > > > OpenStack-dev at lists.openstack.org
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > > The issue is described in the bug that Josh linked
> > > (https://github.com/pypa/pip/issues/988). Basically pip doesn't do
> > > dependency resolution in a way that lets you treat requirements as
> > > order independent. For that to be the case pip would have to
> > > evaluate all dependencies together then install the intersection of those
> dependencies.
> > > Instead it iterates over the list(s) in order and evaluates each
> > > dependency as it is found.
> > >
> > > Your example basically describes where this breaks. You can both
> > > depend on the same dependency at different versions and pip will
> > > install a version that satisfies only one of the dependencies and
> > > not the other leading to a failed install. However I think a more
> > > common case is that openstack will pin a dependency and say
> > > Y>=1.0,<2.0 and the X dependency will say Y>=1.0. If the X
> > > dependency comes first you get version 2.5 which is not valid for your
> specification of Y>=1.0,<2.0 and pip fails.
> > > You fix this by listing Y before X dependency that installs Y with
> > > less restrictive boundaries.
> > >
> > > Another example of a slightly different failure would be hacking,
> > > flake8, pep8, and pyflakes. Hacking installs a specific version of
> > > flake8, pep8, and pyflakes so that we do static lint checking with
> > > consistent checks each release. If you sort this list alphabetically
> > > instead of allowing hacking to install its deps flake8 will come first and you
> can get a different version of pep8.
> > > Different versions of pep8 check different things and now the gate
> > > has broken.
> > >
> > > The most problematic thing is you can't count on your dependencies
> > > from not breaking you if they come first (because they are evaluated
> first).
> > > So in cases where we know order is important (hacking and pbr and
> > > probably a handful of others) we should be listing them as early as
> > > possible in the requirements.
> > >
> > > Clark
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > Thanks Clark,
> >
> > To be honest the issue nor your explanation did clarify this to me.
> > Please forgive me hunting this, but it seems to be extremely important
> > topic so I would like to understand where it's coming from (and
> > hopefully others will benefit from it too).
> >
> > So if I got this right the problem is the dependencies that has not
> > been pinned. Even that issue which has been open is pretty clear about
> that.
> > Also by the pip documentation and the issue text itself if we have
> > those dependencies pinned to our limits on the first level (our
> > requirements.txt) they will be obeyed regardless where they are on the
> > file. Just their dependencies (that are not listed) are not going to
> > be cross checked.
> >
> > Now wouldn't it be right way to attack this problem to actually
> > specify the limits so that we have all of them in the requirements
> > (and they don't break if the maintainer of that first one happens to
> > change the
> > dependencies) rather than trusting the order giving us right
> > dependencies?
> >
> Pinning all transitive dependencies in all OpenStack projects so that we can
> avoid a handful of conflicts is not worth it. OpenStack has a couple hundred
> first level dependencies [0] and it is enough work loosely managing those.
> Conflicts arise when two first level dependencies have similar but conflicting
> second level or more recursive dependencies that conflict. We can avoid
> those conflicts with simple ordering of dependency files which is much easier
> than the alternative.
> 
> I do not see any benefit to a naive sort of requirements files but there is a
> benefit to giving them a specific order so that we can avoid these conflicts.
> 
> It is also worth nothing that finding new failures as they happen is beneficial.
> It is painful, but without it there is very little in the way of advance warning.
> You want to find this stuff out as soon as possible and not at release time.
> 
> [0]
> https://git.openstack.org/cgit/openstack/requirements/tree/global-
> requirements.txt
> 
> Clark
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks for taking time to explaining this to me Clark,

Suddenly it starts to make sense!

- Erno