Open Stack

On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote:
> > -----Original Message-----
> > From: Sean Dague [mailto:sean at dague.net]
> > Sent: 03 September 2014 13:37
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
> > 
> > I'm not sure why people keep showing up with "sort requirements" patches
> > like - https://review.openstack.org/#/c/76817/6, however, they do.
> > 
> > All of these need to be -2ed with predjudice.
> > 
> > requirements.txt is not a declarative interface. The order is important as pip
> > processes it in the order it is. Changing the order has impacts on the overall
> > integration which can cause wedges later.
> > 
> > So please stop.
> > 
> > 	-Sean
> > 
> > --
> > Sean Dague
> > http://dague.net
> >
> 
> Hi Sean & all,
> 
> Could you please open this up a little bit? What are we afraid breaking
> regarding the order of these requirements? I tried to go through pip
> documentation but I could not find reason of specific order of the lines,
> references to keep the order there was 'though.
> 
> I'm now assuming one thing here as I do not know if that's the case. None
> of the packages enables/disables functionality depending of what has been
> installed on the system before, but they have their own dependencies to
> provide those. Based on this assumption I can think of only one scenario
> causing us issues. That is us abusing the example in point 2 of
> https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning;
> we install package X depending on package Y>=1.0,<2.0 before installing
> package Z depending on Y>=1.0 to ensure that package Y<2.0 without
> pinning package Y in our requirements.txt. I certainly hope that this is
> not the case as depending 3rd party vendor providing us specific version
> of dependency package would be extremely stupid.
> 
> Other than that I really don't know how the order could cause us issues,
> but I would be really happy to learn something new today if that is the
> case or if my assumption went wrong.
> 
> Best Regards,
> Erno (jokke_) Kuvaja
>  
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

The issue is described in the bug that Josh linked
(https://github.com/pypa/pip/issues/988). Basically pip doesn't do
dependency resolution in a way that lets you treat requirements as order
independent. For that to be the case pip would have to evaluate all
dependencies together then install the intersection of those
dependencies. Instead it iterates over the list(s) in order and
evaluates each dependency as it is found.

Your example basically describes where this breaks. You can both depend
on the same dependency at different versions and pip will install a
version that satisfies only one of the dependencies and not the other
leading to a failed install. However I think a more common case is that
openstack will pin a dependency and say Y>=1.0,<2.0 and the X dependency
will say Y>=1.0. If the X dependency comes first you get version 2.5
which is not valid for your specification of Y>=1.0,<2.0 and pip fails.
You fix this by listing Y before X dependency that installs Y with less
restrictive boundaries.

Another example of a slightly different failure would be hacking,
flake8, pep8, and pyflakes. Hacking installs a specific version of
flake8, pep8, and pyflakes so that we do static lint checking with
consistent checks each release. If you sort this list alphabetically
instead of allowing hacking to install its deps flake8 will come first
and you can get a different version of pep8. Different versions of pep8
check different things and now the gate has broken.

The most problematic thing is you can't count on your dependencies from
not breaking you if they come first (because they are evaluated first).
So in cases where we know order is important (hacking and pbr and
probably a handful of others) we should be listing them as early as
possible in the requirements.

Clark