[openstack-dev] Requests + urllib3 + distro packages

Robert Collins robertc at robertcollins.net
Fri Oct 9 00:39:42 UTC 2015

This is a bugbear that keeps cropping up and biting us. I'm hoping we
can figure out a permanent fix.

The problem that occurs is the result of a few interacting things:
 - requests has very very specific versions of urllib3 it works with.
So specific they aren't always released yet.

 - Linux vendors often unbundle urllib3 from requests and then apply
what patches were needed to their urllib3; while not updating their
requests package dependencies to reflect this.

 - we use urllib3 in some places and requests in others (but we don't
mix them up)

 - if for any reason we have a distro-altered requests + a
pip-installed urllib3, requests will [usually] break... see the 'not
always released yet' key thing above.

Now, there are lots of places this last thing can happen; they all
depend on us having a dependency on requests that is compatible with
the version installed by the distro, but a urllib3 dependency that
triggers an upgrade of just urllib3. When constraints are in use, the
requests version has to match the distro requests version exactly, but
that will happen from time to time.


 - dvsm test jobs where the base image already has python-requests
installed in it

 - virtualenvs where system-site-packages are enabled

There are a few strategies that have been proposed to fix this. AIUI they are:
 - make sure none of our testing environments include distro requests packages.

 - make our requirements be tightly matched to what requests needs to
deal with the unbundling

 - teach pip how to identify and avoid this situation by always
upgrading requests (even if thats just a re-install of the version
from PyPI) when the installed requests is a distro installed version
**and** urllib3 is being modified.

 - get the distros to stop un-vendoring urllib3

The first one addresses the situation for the CI gate but doesn't
avoid developers getting bitten on their local machines. And
installing any distro thing that uses requests would re-instate the
potential for breakage. So while its not harmful, I don't think its
sufficient to make this go away.

The second is trivially insufficient - anytime requests vendored
urllib3 is not precisely identical to a released urllib3, it becomes
impossible to satisfy that via dependency version pinning - the only
way to satisfy it is with the urllib3 in the distro that has whatever
change was needed included.

The third approach will require some negotiation I suspect - because
its aesthetically wrong: from an upstream perspective urllib3 is
independent of requests, and vice-versa, but from a distro perspective
they are tightly coupled, no variation permitted.

The fourth approach meets the stone wall of 'but security' and 'no
redundancy permitted' - I don't have the energy to try and get through
the near-religious mindset I've encountered there before, though hey -
if Fedora and Debian and Ubuntu folk are all interested in figuring
out a sustainable way forward, that would be great: please don't feel
cut out, I'm just not expecting anything.

If there are other approaches, great - please throw them up here.


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

More information about the OpenStack-dev mailing list