Open Stack

On 10 October 2015 at 02:58, Cory Benfield <cory at lukasa.co.uk> wrote:
>
>> On 9 Oct 2015, at 14:40, William M Edmonds <edmondsw at us.ibm.com> wrote:
>>
>> Cory Benfield <cory at ...> writes:
>> > > The problem that occurs is the result of a few interacting things:
>> > >  - requests has very very specific versions of urllib3 it works with.
>> > > So specific they aren't always released yet.
>> >
>> > This should no longer be true. Our downstream redistributors pointedout to us
>> > that this  was making their lives harder than they needed to be, so it's now
>> > our policy to only  update to actual release versions of urllib3.
>>
>> That's great... except that I'm confused as to why requests would continue to repackage urllib3 if that's the case. Why not just prereq the version of urllib3 that it needs? I thought the one and only answer to that question had been so that requests could package non-standard versions.
>>
>
> That is not and was never the only reason for vendoring urllib3. However, and I cannot stress this enough, the decision to vendor urllib3 is *not going to be changed on this thread*. If and when it changes, it will be by consensus decision from the requests maintenance team, which we do not have at this time.
>
> Further, as I pointed out to Donald Stufft on IRC, if requests unbundled urllib3 *today* that would not fix the problem. The reason is that we’d specify our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note would still cause exactly the problem observed in this thread.

Actually, that would fix the problem (in conjunction with a fix to
https://github.com/pypa/pip/issues/2687 - which is conceptually
trivial once 988 is fixed).

> As you correctly identify in your subsequent email, William, the core problem is mixing of packages from distributions and PyPI. This happens with any tool with external dependencies: if you subsequently install a different version of a dependency using a packaging tool that is not aware of some of the dependency tree, it is entirely plausible that an incompatible version will be installed. It’s not hard to trigger this kind of thing on Ubuntu. IMO, what OpenStack needs is a decision about where it’s getting its packages from, and then to refuse to mix the two.

We can't do that for all our downstreams. Further, Ubuntu preserve
dependency information - I think a key underlying issue is that they
don't fix up the dependency data for requests when they alter it. I've
filed https://bugs.launchpad.net/ubuntu/+source/python-requests/+bug/1505039
to complement the one filed on Fedora earlier in this thread.

*We* have the privilege of working directly with folk like libvirt
that have been problematic in the past and getting those things
addressed, so that we can run in a virtualenv happily. But we can't
insist that user X who want to use some openstack library that uses
requests but also some other thing (maybe some SWIG binding or
something). So framing this as being driven by the mix is false: the
thing that drives this is a combination - e.g. coming in part from
defects in pip, and the existence of things that can't be installed in
virtualenvs.

Obviously a trivial workaround is to always use virtualenvs and not
system-site-packages.

To sum up the thread, it sounds to me like a viable way-forward is:

 - get distros to fixup their requests Python dependencies (and
hopefully they can update that in stable releases).
 - fix the existing known bugs in pip where such accurate dependencies
are violated by some operations.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud