Open Stack

On Wed, Sep 3, 2014, at 01:06 PM, Kuvaja, Erno wrote:
> > -----Original Message-----
> > From: Clark Boylan [mailto:cboylan at sapwetik.org]
> > Sent: 03 September 2014 20:10
> > To: openstack-dev at lists.openstack.org
> > Subject: Re: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
> > 
> > 
> > 
> > On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote:
> > > > -----Original Message-----
> > > > From: Sean Dague [mailto:sean at dague.net]
> > > > Sent: 03 September 2014 13:37
> > > > To: OpenStack Development Mailing List (not for usage questions)
> > > > Subject: [openstack-dev] [all] [glance] do NOT ever sort
> > > > requirements.txt
> > > >
> > > > I'm not sure why people keep showing up with "sort requirements"
> > > > patches like - https://review.openstack.org/#/c/76817/6, however, they
> > do.
> > > >
> > > > All of these need to be -2ed with predjudice.
> > > >
> > > > requirements.txt is not a declarative interface. The order is
> > > > important as pip processes it in the order it is. Changing the order
> > > > has impacts on the overall integration which can cause wedges later.
> > > >
> > > > So please stop.
> > > >
> > > > 	-Sean
> > > >
> > > > --
> > > > Sean Dague
> > > > http://dague.net
> > > >
> > >
> > > Hi Sean & all,
> > >
> > > Could you please open this up a little bit? What are we afraid
> > > breaking regarding the order of these requirements? I tried to go
> > > through pip documentation but I could not find reason of specific
> > > order of the lines, references to keep the order there was 'though.
> > >
> > > I'm now assuming one thing here as I do not know if that's the case.
> > > None of the packages enables/disables functionality depending of what
> > > has been installed on the system before, but they have their own
> > > dependencies to provide those. Based on this assumption I can think of
> > > only one scenario causing us issues. That is us abusing the example in
> > > point 2 of
> > > https://pip.pypa.io/en/latest/user_guide.html#requirements-files
> > > meaning; we install package X depending on package Y>=1.0,<2.0 before
> > > installing package Z depending on Y>=1.0 to ensure that package Y<2.0
> > > without pinning package Y in our requirements.txt. I certainly hope
> > > that this is not the case as depending 3rd party vendor providing us specific
> > version of dependency package would be extremely stupid.
> > >
> > > Other than that I really don't know how the order could cause us
> > > issues, but I would be really happy to learn something new today if
> > > that is the case or if my assumption went wrong.
> > >
> > > Best Regards,
> > > Erno (jokke_) Kuvaja
> > >
> > > > _______________________________________________
> > > > OpenStack-dev mailing list
> > > > OpenStack-dev at lists.openstack.org
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > The issue is described in the bug that Josh linked
> > (https://github.com/pypa/pip/issues/988). Basically pip doesn't do
> > dependency resolution in a way that lets you treat requirements as order
> > independent. For that to be the case pip would have to evaluate all
> > dependencies together then install the intersection of those dependencies.
> > Instead it iterates over the list(s) in order and evaluates each dependency as
> > it is found.
> > 
> > Your example basically describes where this breaks. You can both depend on
> > the same dependency at different versions and pip will install a version that
> > satisfies only one of the dependencies and not the other leading to a failed
> > install. However I think a more common case is that openstack will pin a
> > dependency and say Y>=1.0,<2.0 and the X dependency will say Y>=1.0. If
> > the X dependency comes first you get version 2.5 which is not valid for your
> > specification of Y>=1.0,<2.0 and pip fails.
> > You fix this by listing Y before X dependency that installs Y with less restrictive
> > boundaries.
> > 
> > Another example of a slightly different failure would be hacking, flake8,
> > pep8, and pyflakes. Hacking installs a specific version of flake8, pep8, and
> > pyflakes so that we do static lint checking with consistent checks each
> > release. If you sort this list alphabetically instead of allowing hacking to install
> > its deps flake8 will come first and you can get a different version of pep8.
> > Different versions of pep8 check different things and now the gate has
> > broken.
> > 
> > The most problematic thing is you can't count on your dependencies from
> > not breaking you if they come first (because they are evaluated first).
> > So in cases where we know order is important (hacking and pbr and probably
> > a handful of others) we should be listing them as early as possible in the
> > requirements.
> > 
> > Clark
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> Thanks Clark,
> 
> To be honest the issue nor your explanation did clarify this to me.
> Please forgive me hunting this, but it seems to be extremely important
> topic so I would like to understand where it's coming from (and hopefully
> others will benefit from it too). 
> 
> So if I got this right the problem is the dependencies that has not been
> pinned. Even that issue which has been open is pretty clear about that.
> Also by the pip documentation and the issue text itself if we have those
> dependencies pinned to our limits on the first level (our
> requirements.txt) they will be obeyed regardless where they are on the
> file. Just their dependencies (that are not listed) are not going to be
> cross checked.
> 
> Now wouldn't it be right way to attack this problem to actually specify
> the limits so that we have all of them in the requirements (and they
> don't break if the maintainer of that first one happens to change the
> dependencies) rather than trusting the order giving us right
> dependencies?
> 
Pinning all transitive dependencies in all OpenStack projects so that we
can avoid a handful of conflicts is not worth it. OpenStack has a couple
hundred first level dependencies [0] and it is enough work loosely
managing those. Conflicts arise when two first level dependencies have
similar but conflicting second level or more recursive dependencies that
conflict. We can avoid those conflicts with simple ordering of
dependency files which is much easier than the alternative.

I do not see any benefit to a naive sort of requirements files but there
is a benefit to giving them a specific order so that we can avoid these
conflicts.

It is also worth nothing that finding new failures as they happen is
beneficial. It is painful, but without it there is very little in the
way of advance warning. You want to find this stuff out as soon as
possible and not at release time.

[0]
https://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt

Clark