Open Stack

On 09/03/2014 09:09 PM, Clark Boylan wrote:
> 
> 
> On Wed, Sep 3, 2014, at 11:51 AM, Kuvaja, Erno wrote:
>>> -----Original Message-----
>>> From: Sean Dague [mailto:sean at dague.net]
>>> Sent: 03 September 2014 13:37
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Subject: [openstack-dev] [all] [glance] do NOT ever sort requirements.txt
>>>
>>> I'm not sure why people keep showing up with "sort requirements" patches
>>> like - https://review.openstack.org/#/c/76817/6, however, they do.
>>>
>>> All of these need to be -2ed with predjudice.
>>>
>>> requirements.txt is not a declarative interface. The order is important as pip
>>> processes it in the order it is. Changing the order has impacts on the overall
>>> integration which can cause wedges later.
>>>
>>> So please stop.
>>>
>>> 	-Sean
>>>
>>> --
>>> Sean Dague
>>> http://dague.net
>>>
>>
>> Hi Sean & all,
>>
>> Could you please open this up a little bit? What are we afraid breaking
>> regarding the order of these requirements? I tried to go through pip
>> documentation but I could not find reason of specific order of the lines,
>> references to keep the order there was 'though.
>>
>> I'm now assuming one thing here as I do not know if that's the case. None
>> of the packages enables/disables functionality depending of what has been
>> installed on the system before, but they have their own dependencies to
>> provide those. Based on this assumption I can think of only one scenario
>> causing us issues. That is us abusing the example in point 2 of
>> https://pip.pypa.io/en/latest/user_guide.html#requirements-files meaning;
>> we install package X depending on package Y>=1.0,<2.0 before installing
>> package Z depending on Y>=1.0 to ensure that package Y<2.0 without
>> pinning package Y in our requirements.txt. I certainly hope that this is
>> not the case as depending 3rd party vendor providing us specific version
>> of dependency package would be extremely stupid.
>>
>> Other than that I really don't know how the order could cause us issues,
>> but I would be really happy to learn something new today if that is the
>> case or if my assumption went wrong.
>>
>> Best Regards,
>> Erno (jokke_) Kuvaja
>>  
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> The issue is described in the bug that Josh linked
> (https://github.com/pypa/pip/issues/988). Basically pip doesn't do
> dependency resolution in a way that lets you treat requirements as order
> independent. For that to be the case pip would have to evaluate all
> dependencies together then install the intersection of those
> dependencies. Instead it iterates over the list(s) in order and
> evaluates each dependency as it is found.
> 
> Your example basically describes where this breaks. You can both depend
> on the same dependency at different versions and pip will install a
> version that satisfies only one of the dependencies and not the other
> leading to a failed install. However I think a more common case is that
> openstack will pin a dependency and say Y>=1.0,<2.0 and the X dependency
> will say Y>=1.0. If the X dependency comes first you get version 2.5
> which is not valid for your specification of Y>=1.0,<2.0 and pip fails.
> You fix this by listing Y before X dependency that installs Y with less
> restrictive boundaries.
> 
> Another example of a slightly different failure would be hacking,
> flake8, pep8, and pyflakes. Hacking installs a specific version of
> flake8, pep8, and pyflakes so that we do static lint checking with
> consistent checks each release. If you sort this list alphabetically
> instead of allowing hacking to install its deps flake8 will come first
> and you can get a different version of pep8. Different versions of pep8
> check different things and now the gate has broken.
> 
> The most problematic thing is you can't count on your dependencies from
> not breaking you if they come first (because they are evaluated first).
> So in cases where we know order is important (hacking and pbr and
> probably a handful of others) we should be listing them as early as
> possible in the requirements.


So, is there a specific order to look out for?

AFAIU requirements should have pbr as first requirement and
test-requirements should have hacking as first one. Is there anything
else? What's the best place to document this?

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)
    GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126