Open Stack

On 10/12/2015 02:16 AM, Robert Collins wrote:
> On 10 October 2015 at 02:58, Cory Benfield <cory at lukasa.co.uk> wrote:
>>
>>> On 9 Oct 2015, at 14:40, William M Edmonds <edmondsw at us.ibm.com> wrote:
>>>
>>> Cory Benfield <cory at ...> writes:
>>>>> The problem that occurs is the result of a few interacting things:
>>>>>  - requests has very very specific versions of urllib3 it works with.
>>>>> So specific they aren't always released yet.
>>>>
>>>> This should no longer be true. Our downstream redistributors pointedout to us
>>>> that this  was making their lives harder than they needed to be, so it's now
>>>> our policy to only  update to actual release versions of urllib3.
>>>
>>> That's great... except that I'm confused as to why requests would continue to repackage urllib3 if that's the case. Why not just prereq the version of urllib3 that it needs? I thought the one and only answer to that question had been so that requests could package non-standard versions.
>>>
>>
>> That is not and was never the only reason for vendoring urllib3. However, and I cannot stress this enough, the decision to vendor urllib3 is *not going to be changed on this thread*. If and when it changes, it will be by consensus decision from the requests maintenance team, which we do not have at this time.
>>
>> Further, as I pointed out to Donald Stufft on IRC, if requests unbundled urllib3 *today* that would not fix the problem. The reason is that we’d specify our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note would still cause exactly the problem observed in this thread.
> 
> Actually, that would fix the problem (in conjunction with a fix to
> https://github.com/pypa/pip/issues/2687 - which is conceptually
> trivial once 988 is fixed).
> 
>> As you correctly identify in your subsequent email, William, the core problem is mixing of packages from distributions and PyPI. This happens with any tool with external dependencies: if you subsequently install a different version of a dependency using a packaging tool that is not aware of some of the dependency tree, it is entirely plausible that an incompatible version will be installed. It’s not hard to trigger this kind of thing on Ubuntu. IMO, what OpenStack needs is a decision about where it’s getting its packages from, and then to refuse to mix the two.
> 
> We can't do that for all our downstreams. Further, Ubuntu preserve
> dependency information - I think a key underlying issue is that they
> don't fix up the dependency data for requests when they alter it. I've
> filed https://bugs.launchpad.net/ubuntu/+source/python-requests/+bug/1505039
> to complement the one filed on Fedora earlier in this thread.

Well, since Ubuntu uses the Debian package simply by syncing from
Debian, I would suggest to file a bug against the Debian package instead
[1].

> Obviously a trivial workaround is to always use virtualenvs and not
> system-site-packages.

or opposite way... always use system-site-packages! :)

Has the infra team ever thought about doing that for (at least) all of
the 3rd party libs we use? I'd love to work closer with the infra team
to provide them with missing packages they would need, and I'm sure my
RPM buddy Haikel would too. This also would help getting our
openstack/{deb,rpm}- projects up to speed as well.

> To sum up the thread, it sounds to me like a viable way-forward is:
> 
>  - get distros to fixup their requests Python dependencies (and
> hopefully they can update that in stable releases).

The maintainer for both urllib3 and requests is a single person, so I'm
quite sure he is aware of the issue, and make sure that both packages
are compatible. Otherwise, opening bugs in Debian [1] to have him fix it
would obviously work. I don't believe that adding version upper-bounds
in the deb package would solve anything, we just need to not make
incompatible versions co-exist at a given moment in the distro.

Cheers,

Thomas Goirand (zigo)

[1] https://www.debian.org/Bugs/Reporting (note: I know Robert knows how
to file a bug in Debian, so this is obviously not for him that I'm
giving this URL... :) )