[openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

Ian Wienand iwienand at redhat.com
Tue Aug 6 01:37:53 UTC 2013


On Mon, Aug 05, 2013 at 12:03:07PM -0500, Dean Troyer wrote:
> * proposals to use a tool to automatically decide between package and
> PyPI (harlowja, sdague):  this works well on the surface, but anything
> that does not take in to account the dependencies in these packages
> going BOTH ways is going to fail.  For example: on RHEL6 setuptools is
> 0.6.10, we want 0.9.8 (the merged release w/distribute).  Removing
> python-setuptools will also remove python-nose, numpy and other
> packages depending on what is installed.  So fine, those can be
> re-installed with pip.  But a) we don't want to rebuild numpy (just
> bear with me here), and b) the packaged python-nose 0.10.4 meets the
> version requirement in requirements.txt so the package will be
> re-installed, bringing with it python-setuptools 0.6.10 overwriting
> the pip installation of 0.9.8.

I think Anvil is working with the package management system so that
scenario doesn't happen.  The "fine, those can be re-installed with
pip" bit is where the problem occurs.

The Anvil process is, as I understand it:

 1) parse requirements.txt
 2) see what can be satisfied via yum
 3) pip download the rest
 4) remove downloaded dependencies that are satisfied by yum
 5) make rpms of now remaining pip downloads
 6) install the *whole lot*

The "whole lot" bit is important, because you can't have conflicts
there.  Say requirements.txt brings in setuptools-0.9.8; Anvil will
create a python-setuptools 0.9.8 package.  If rpm-packaged nose relies
*exactly* python-setuptools at 0.6.10, there will be a conflict -- I
think the installation would fail to complete.  But likely, that
package doesn't care and gets it dep satisfied by 0.9.8 [1]

Because you're not using pip to install directly, you don't have this
confusion around who owns files in /usr/lib/python2.6/site-packages
and have rpm or pip overwriting each other -- RPM owns all the files
and that's that.

> Removing python-setuptools will also remove python-nose, numpy and
> other packages depending on what is installed.

Nowhere is python-setuptools removed; just upgraded.

Recently trying out Anvil, it seems to have the correct solution to my
mind.

-i

[1] From a quick look at Anvil I don't think it would handle this
situation, which is probably unsolvable (if a rpm-package wants one
version and devstack wants another, and it all has to be in
/usr/lib/python2.6/site-packages, then *someone* is going to lose).
But I don't think <= or == dependencies in rpms are too common, so you
can just drop in the new version and hope it remains backwards
compatible :)



More information about the OpenStack-dev mailing list