Open Stack

On 10/08/2015 07:39 PM, Robert Collins wrote:
> This is a bugbear that keeps cropping up and biting us. I'm hoping we
> can figure out a permanent fix.
> 
> The problem that occurs is the result of a few interacting things:
>  - requests has very very specific versions of urllib3 it works with.
> So specific they aren't always released yet.
> 
>  - Linux vendors often unbundle urllib3 from requests and then apply
> what patches were needed to their urllib3; while not updating their
> requests package dependencies to reflect this.
> 
>  - we use urllib3 in some places and requests in others (but we don't
> mix them up)
> 
>  - if for any reason we have a distro-altered requests + a
> pip-installed urllib3, requests will [usually] break... see the 'not
> always released yet' key thing above.
> 
> Now, there are lots of places this last thing can happen; they all
> depend on us having a dependency on requests that is compatible with
> the version installed by the distro, but a urllib3 dependency that
> triggers an upgrade of just urllib3. When constraints are in use, the
> requests version has to match the distro requests version exactly, but
> that will happen from time to time.
> 
> e.g.
> 
>  - dvsm test jobs where the base image already has python-requests
> installed in it
> 
>  - virtualenvs where system-site-packages are enabled
> 
> 
> There are a few strategies that have been proposed to fix this. AIUI they are:
>  - make sure none of our testing environments include distro requests packages.
> 
>  - make our requirements be tightly matched to what requests needs to
> deal with the unbundling
> 
>  - teach pip how to identify and avoid this situation by always
> upgrading requests (even if thats just a re-install of the version
> from PyPI) when the installed requests is a distro installed version
> **and** urllib3 is being modified.
> 
>  - get the distros to stop un-vendoring urllib3
> 
> 
> The first one addresses the situation for the CI gate but doesn't
> avoid developers getting bitten on their local machines. And
> installing any distro thing that uses requests would re-instate the
> potential for breakage. So while its not harmful, I don't think its
> sufficient to make this go away.
> 
> The second is trivially insufficient - anytime requests vendored
> urllib3 is not precisely identical to a released urllib3, it becomes
> impossible to satisfy that via dependency version pinning - the only
> way to satisfy it is with the urllib3 in the distro that has whatever
> change was needed included.
> 
> The third approach will require some negotiation I suspect - because
> its aesthetically wrong: from an upstream perspective urllib3 is
> independent of requests, and vice-versa, but from a distro perspective
> they are tightly coupled, no variation permitted.
> 
> The fourth approach meets the stone wall of 'but security' and 'no
> redundancy permitted' - I don't have the energy to try and get through
> the near-religious mindset I've encountered there before, though hey -
> if Fedora and Debian and Ubuntu folk are all interested in figuring
> out a sustainable way forward, that would be great: please don't feel
> cut out, I'm just not expecting anything.
> 
> If there are other approaches, great - please throw them up here.
> 
> -Rob
> 

At least for my packaging for Gentoo I haven't run into any issues here.
We have multiple versions of both requests and urllib3 available and I
can add a missing version if needed.

-- 
Matthew Thode (prometheanfire)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151013/5baf0cb8/attachment.pgp>