Open Stack

Hi,

When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has
been specified, this function tries to also install the development library
for Python 2.x, I guess just in case some package has not declared proper
Python 3 support or something. It then proceeds to install the Python 3
version of the library and all its dependencies.

Unfortunately there is a problem with that, and specifically with script
files installed in the system's executable files directory, e.g.
/usr/local/bin. The problem appears when some Python library has already
been installed for Python 3 (and has installed its script files), but is
now installed for Python 2 (overwriting the script files) and is then not
forcefully reinstalled for Python 3, since it is already present. Thus, the
script files are last modified by the Python 2 library installation and
they have a hashbang line saying `python2.x` - so if something then tries
to execute them, they will run and use modules and libraries for Python 2
only.

We experienced this problem when running the cinderlib tests from Cinder's
`playbooks/cinderlib-run.yaml` file - it finds a unit2 executable
(installed by the unittest2 library) and runs it, hoping that unit2 will be
able to discover and collect the cinderlib tests and load the cinderlib
modules. However, since unittest2 has last been installed as a Python 2
library, unit2 runs with Python 2 and fails to locate the cinderlib
modules. (Yes, we know that there are other ways to run the cinderlib
tests; this message is about the problem exposed by this way of running
them)

The obvious solution would be to instruct the Python 2 pip to not install
script (or other shared) files at all; unfortunately,
https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on
install"), detailing a very similar use case ("need it installed for Python
2, but want to use it with Python 3") has been open for almost exactly
three years now with no progress. I wonder if I could try to help, but even
if this issue is resolved, there will be some time before OpenStack can
actually depend on a recent enough version of pip.

A horrible workaround would be to find the binary directory before
installing the Python 2 library (using something like `pip3.7 show
somepackage` and then running some heuristics on the "Location" field),
tar'ing it up and then restoring it... but I don't know if I even want to
think about this.

Another possible way forward would be to consider whether we still want the
Python 2 libraries installed - is OpenStack's Python 3 transition reached a
far enough stage to assume that any projects that still require Python 2
*and* fail to declare their Python 2 dependencies properly are buggy? To be
honest, this seems the most reasonable path for me - drop the "also install
the Python 2 libs" code and see what happens. I could try to make this
change in a couple of test runs in our third-party Cinder CI system and see
if something breaks.

Here is a breakdown of what happens, with links to the log of the StorPool
third-party CI system for Cinder:

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_43_55_691087
`stack.sh` invokes `pip_install` for `os-testr`

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_43_56_030839
`pip_install` sees that we want a Python 3 installation and invokes
`pip3.7` to install os-testr.

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_43_59_869198
`pip3.7` wants to install `unittest2`

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_44_15_851337
`pip3.7` has installed `unittest2` - now `/usr/local/bin/unit2` has a
hashbang line saying `python3.7`

Now this is where it gets, uhm, interesting:

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_45_59_708737
`setup_dev_lib` is invoked for `os-brick`

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_45_59_723318
`setup_dev_lib`, seeing that we really want a Python 3 installation,
decides to install `os-brick` for Python 2 just in case.

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_46_00_661346
`pip2.7` is invoked to install `os-brick` and its dependencies.

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_46_25_209365
`pip2.7` decides it wants to install `unittest2`, too.

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_47_20_924559
`pip2.7` has installed `unittest2`, and now `/usr/local/bin/unit2` has a
hasbang line saying `python2.7`

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_47_21_591114
`setup_dev_lib` turns the Python 3 flag back on.

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_47_22_659564
`pip3.7` is invoked to install `os-brick`

https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempest/82fb46b/job-output.txt.gz#_2019-09-09_05_47_36_759583
`pip3.7` decides (correctly) that it has already installed `unittest2`, so
(only partially correctly) it does not need to install it again.

Thus `/usr/local/bin/unit2` is left with a hashbang line saying `python2.7`.

Thanks for reading this far, I guess :)

G'luck,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190909/0dc235a1/attachment.html>