[devstack][qa][python3] "also install the Python 2 dev library" - still needed?
Hi,
When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has been specified, this function tries to also install the development library for Python 2.x, I guess just in case some package has not declared proper Python 3 support or something. It then proceeds to install the Python 3 version of the library and all its dependencies.
Unfortunately there is a problem with that, and specifically with script files installed in the system's executable files directory, e.g. /usr/local/bin. The problem appears when some Python library has already been installed for Python 3 (and has installed its script files), but is now installed for Python 2 (overwriting the script files) and is then not forcefully reinstalled for Python 3, since it is already present. Thus, the script files are last modified by the Python 2 library installation and they have a hashbang line saying `python2.x` - so if something then tries to execute them, they will run and use modules and libraries for Python 2 only.
We experienced this problem when running the cinderlib tests from Cinder's `playbooks/cinderlib-run.yaml` file - it finds a unit2 executable (installed by the unittest2 library) and runs it, hoping that unit2 will be able to discover and collect the cinderlib tests and load the cinderlib modules. However, since unittest2 has last been installed as a Python 2 library, unit2 runs with Python 2 and fails to locate the cinderlib modules. (Yes, we know that there are other ways to run the cinderlib tests; this message is about the problem exposed by this way of running them)
The obvious solution would be to instruct the Python 2 pip to not install script (or other shared) files at all; unfortunately, https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on install"), detailing a very similar use case ("need it installed for Python 2, but want to use it with Python 3") has been open for almost exactly three years now with no progress. I wonder if I could try to help, but even if this issue is resolved, there will be some time before OpenStack can actually depend on a recent enough version of pip.
A horrible workaround would be to find the binary directory before installing the Python 2 library (using something like `pip3.7 show somepackage` and then running some heuristics on the "Location" field), tar'ing it up and then restoring it... but I don't know if I even want to think about this.
Another possible way forward would be to consider whether we still want the Python 2 libraries installed - is OpenStack's Python 3 transition reached a far enough stage to assume that any projects that still require Python 2 *and* fail to declare their Python 2 dependencies properly are buggy? To be honest, this seems the most reasonable path for me - drop the "also install the Python 2 libs" code and see what happens. I could try to make this change in a couple of test runs in our third-party Cinder CI system and see if something breaks.
Here is a breakdown of what happens, with links to the log of the StorPool third-party CI system for Cinder:
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `stack.sh` invokes `pip_install` for `os-testr`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip_install` sees that we want a Python 3 installation and invokes `pip3.7` to install os-testr.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip3.7` wants to install `unittest2`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip3.7` has installed `unittest2` - now `/usr/local/bin/unit2` has a hashbang line saying `python3.7`
Now this is where it gets, uhm, interesting:
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `setup_dev_lib` is invoked for `os-brick`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `setup_dev_lib`, seeing that we really want a Python 3 installation, decides to install `os-brick` for Python 2 just in case.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip2.7` is invoked to install `os-brick` and its dependencies.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip2.7` decides it wants to install `unittest2`, too.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip2.7` has installed `unittest2`, and now `/usr/local/bin/unit2` has a hasbang line saying `python2.7`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `setup_dev_lib` turns the Python 3 flag back on.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip3.7` is invoked to install `os-brick`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe... `pip3.7` decides (correctly) that it has already installed `unittest2`, so (only partially correctly) it does not need to install it again.
Thus `/usr/local/bin/unit2` is left with a hashbang line saying `python2.7`.
Thanks for reading this far, I guess :)
G'luck, Peter
On Mon, 2019-09-09 at 11:22 +0300, Peter Penchev wrote:
Hi,
When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has been specified, this function tries to also install the development library for Python 2.x, I guess just in case some package has not declared proper Python 3 support or something. It then proceeds to install the Python 3 version of the library and all its dependencies.
Unfortunately there is a problem with that, and specifically with script files installed in the system's executable files directory, e.g. /usr/local/bin. The problem appears when some Python library has already been installed for Python 3 (and has installed its script files), but is now installed for Python 2 (overwriting the script files) and is then not forcefully reinstalled for Python 3, since it is already present. Thus, the script files are last modified by the Python 2 library installation and they have a hashbang line saying `python2.x` - so if something then tries to execute them, they will run and use modules and libraries for Python 2 only.
yes this is a long standing issue. we discovered it a year ago but it was never fix.
in Ussrui i guess one of the first changes to devstack to make it python 3 only will be to chagne that behavior. im not sure if we will be able to change it before then.
whenever you us libs_from_git in your local.conf on a python 3 install it will install them twice both with python 2 and python 3. i hope more distros elect to symlink /usr/bin/python to python 3 some distros have chosen to do that on systems that are python only and i believe that is the correct approch.
when i encountered this it was always resuliting on the script header being #!/usr/bin/python with no version suffix i gues on a system where that points to python 3 the python 2.7 install might write python2.7 there instead?
We experienced this problem when running the cinderlib tests from Cinder's `playbooks/cinderlib-run.yaml` file - it finds a unit2 executable (installed by the unittest2 library) and runs it, hoping that unit2 will be able to discover and collect the cinderlib tests and load the cinderlib modules. However, since unittest2 has last been installed as a Python 2 library, unit2 runs with Python 2 and fails to locate the cinderlib modules. (Yes, we know that there are other ways to run the cinderlib tests; this message is about the problem exposed by this way of running them)
The obvious solution would be to instruct the Python 2 pip to not install script (or other shared) files at all; unfortunately, https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on install"), detailing a very similar use case ("need it installed for Python 2, but want to use it with Python 3") has been open for almost exactly three years now with no progress. I wonder if I could try to help, but even if this issue is resolved, there will be some time before OpenStack can actually depend on a recent enough version of pip.
well the obvious solution is to stop doing this entirly. it was added as a hack to ensure if you use LIB_FROM_GIT in you local.conf that those libs would always be install from the git checkout that you specified in you local.conf for train we are technically requireing all project to run under python 3 so we could remove the fallback mechanium of in stalling under python 2. it was there incase a service installed under python 2 to ensure it used the same version of the lib and did not use a version form pypi instead. i wanted to stop doing this last year but we could not becase not all project could run under python 3. but now that they should be able to we dont need this hack anymore. we should change it to respec the python version you have selected. that will speed up stacking speed as we wont have to install everything twice and fix the issue you have encountered.
A horrible workaround would be to find the binary directory before installing the Python 2 library (using something like `pip3.7 show somepackage` and then running some heuristics on the "Location" field), tar'ing it up and then restoring it... but I don't know if I even want to think about this.
Another possible way forward would be to consider whether we still want the Python 2 libraries installed - is OpenStack's Python 3 transition reached a far enough stage to assume that any projects that still require Python 2 *and* fail to declare their Python 2 dependencies properly are buggy? To be honest, this seems the most reasonable path for me - drop the "also install the Python 2 libs" code and see what happens. I could try to make this change in a couple of test runs in our third-party Cinder CI system and see if something breaks.
Here is a breakdown of what happens, with links to the log of the StorPool third-party CI system for Cinder:
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`stack.sh` invokes `pip_install` for `os-testr`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip_install` sees that we want a Python 3 installation and invokes `pip3.7` to install os-testr.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip3.7` wants to install `unittest2`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip3.7` has installed `unittest2` - now `/usr/local/bin/unit2` has a hashbang line saying `python3.7`
Now this is where it gets, uhm, interesting:
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`setup_dev_lib` is invoked for `os-brick`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`setup_dev_lib`, seeing that we really want a Python 3 installation, decides to install `os-brick` for Python 2 just in case.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip2.7` is invoked to install `os-brick` and its dependencies.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip2.7` decides it wants to install `unittest2`, too.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip2.7` has installed `unittest2`, and now `/usr/local/bin/unit2` has a hasbang line saying `python2.7`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`setup_dev_lib` turns the Python 3 flag back on.
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip3.7` is invoked to install `os-brick`
https://spfactory.storpool.com/logs/80/639180/35/check/cinder-storpool-tempe...
`pip3.7` decides (correctly) that it has already installed `unittest2`, so (only partially correctly) it does not need to install it again.
Thus `/usr/local/bin/unit2` is left with a hashbang line saying `python2.7`.
Thanks for reading this far, I guess :)
G'luck, Peter
On 2019-09-09 12:54:58 +0100 (+0100), Sean Mooney wrote: [...]
i hope more distros elect to symlink /usr/bin/python to python 3 some distros have chosen to do that on systems that are python only and i believe that is the correct approch.
I personally hope they don't, and at least my preferred Linux distro is not planning to do that any time in the foreseeable future (if ever). I see python and python3 as distinct programming languages with their own interpreters, and so any distro which by default pretends that its python3 interpreter is a python interpreter (by claiming the unversioned "python" executable name in the system context search path) is simply broken.
when i encountered this it was always resuliting on the script header being #!/usr/bin/python with no version suffix i gues on a system where that points to python 3 the python 2.7 install might write python2.7 there instead?
Yes, the correct solution is to update those to #!/usr/bin/python3 because at least some distros are going to cease providing a /usr/bin/python executable at all when they drop their 2.7 packages.
On Mon, Sep 9, 2019, at 1:22 AM, Peter Penchev wrote:
Hi,
When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has been specified, this function tries to also install the development library for Python 2.x, I guess just in case some package has not declared proper Python 3 support or something. It then proceeds to install the Python 3 version of the library and all its dependencies.
Unfortunately there is a problem with that, and specifically with script files installed in the system's executable files directory, e.g. /usr/local/bin. The problem appears when some Python library has already been installed for Python 3 (and has installed its script files), but is now installed for Python 2 (overwriting the script files) and is then not forcefully reinstalled for Python 3, since it is already present. Thus, the script files are last modified by the Python 2 library installation and they have a hashbang line saying `python2.x`
- so if something then tries to execute them, they will run and use
modules and libraries for Python 2 only.
We experienced this problem when running the cinderlib tests from Cinder's `playbooks/cinderlib-run.yaml` file - it finds a unit2 executable (installed by the unittest2 library) and runs it, hoping that unit2 will be able to discover and collect the cinderlib tests and load the cinderlib modules. However, since unittest2 has last been installed as a Python 2 library, unit2 runs with Python 2 and fails to locate the cinderlib modules. (Yes, we know that there are other ways to run the cinderlib tests; this message is about the problem exposed by this way of running them)
One option here is to explicitly run the file under the python version you want. I do this with `pbr freeze` frequently to ensure I'm looking at the correct version of software for the correct version of python. For example:
python3 /usr/local/bin/pbr freeze | grep $packagename python2 /usr/local/bin/pbr freeze | grep $packagename
Then as long as you have installed the utility (in my case pbr) under both python versions it should just work assuming they don't write different files for different versions of python at install time.
The obvious solution would be to instruct the Python 2 pip to not install script (or other shared) files at all; unfortunately, https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on install"), detailing a very similar use case ("need it installed for Python 2, but want to use it with Python 3") has been open for almost exactly three years now with no progress. I wonder if I could try to help, but even if this issue is resolved, there will be some time before OpenStack can actually depend on a recent enough version of pip.
Note OpenStack tests with, and as a result possibly requires, the latest version of pip. Fixing this in pip shouldn't be a problem as long as they make a release not long after.
A horrible workaround would be to find the binary directory before installing the Python 2 library (using something like `pip3.7 show somepackage` and then running some heuristics on the "Location" field), tar'ing it up and then restoring it... but I don't know if I even want to think about this.
Another possible way forward would be to consider whether we still want the Python 2 libraries installed - is OpenStack's Python 3 transition reached a far enough stage to assume that any projects that still require Python 2 *and* fail to declare their Python 2 dependencies properly are buggy? To be honest, this seems the most reasonable path for me - drop the "also install the Python 2 libs" code and see what happens. I could try to make this change in a couple of test runs in our third-party Cinder CI system and see if something breaks.
snip
Hope this helps, Clark
On Mon, Sep 9, 2019 at 7:42 PM Clark Boylan cboylan@sapwetik.org wrote:
On Mon, Sep 9, 2019, at 1:22 AM, Peter Penchev wrote:
Hi,
When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has been specified, this function tries to also install the development library for Python 2.x, I guess just in case some package has not declared proper Python 3 support or something. It then proceeds to install the Python 3 version of the library and all its dependencies.
Unfortunately there is a problem with that, and specifically with script files installed in the system's executable files directory, e.g. /usr/local/bin. The problem appears when some Python library has already been installed for Python 3 (and has installed its script files), but is now installed for Python 2 (overwriting the script files) and is then not forcefully reinstalled for Python 3, since it is already present. Thus, the script files are last modified by the Python 2 library installation and they have a hashbang line saying `python2.x`
- so if something then tries to execute them, they will run and use
modules and libraries for Python 2 only.
We experienced this problem when running the cinderlib tests from Cinder's `playbooks/cinderlib-run.yaml` file - it finds a unit2 executable (installed by the unittest2 library) and runs it, hoping that unit2 will be able to discover and collect the cinderlib tests and load the cinderlib modules. However, since unittest2 has last been installed as a Python 2 library, unit2 runs with Python 2 and fails to locate the cinderlib modules. (Yes, we know that there are other ways to run the cinderlib tests; this message is about the problem exposed by this way of running them)
One option here is to explicitly run the file under the python version you want. I do this with `pbr freeze` frequently to ensure I'm looking at the correct version of software for the correct version of python. For example:
python3 /usr/local/bin/pbr freeze | grep $packagename python2 /usr/local/bin/pbr freeze | grep $packagename
Then as long as you have installed the utility (in my case pbr) under both python versions it should just work assuming they don't write different files for different versions of python at install time.
This is what we ended up doing (sorry, I might have mentioned that in the original message; it was a solved problem for our CI) - we modified the Ansible job to explicitly run "python3.7 unit2". So, yeah, my message was more to point out the general problem than to ask for help for our specific case, but still, yeah, thanks, that's exactly what we did.
The obvious solution would be to instruct the Python 2 pip to not install script (or other shared) files at all; unfortunately, https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on install"), detailing a very similar use case ("need it installed for Python 2, but want to use it with Python 3") has been open for almost exactly three years now with no progress. I wonder if I could try to help, but even if this issue is resolved, there will be some time before OpenStack can actually depend on a recent enough version of pip.
Note OpenStack tests with, and as a result possibly requires, the latest version of pip. Fixing this in pip shouldn't be a problem as long as they make a release not long after.
Right, I did briefly wonder whether this was true while writing my mail, I should have taken the time to check and see that devstack actually installs its own version of pip and removes any versions installed by OS packages. Hm, I just might try my hand at that in the coming days or weeks, but I can't really make any promises.
A horrible workaround would be to find the binary directory before installing the Python 2 library (using something like `pip3.7 show somepackage` and then running some heuristics on the "Location" field), tar'ing it up and then restoring it... but I don't know if I even want to think about this.
Another possible way forward would be to consider whether we still want the Python 2 libraries installed - is OpenStack's Python 3 transition reached a far enough stage to assume that any projects that still require Python 2 *and* fail to declare their Python 2 dependencies properly are buggy? To be honest, this seems the most reasonable path for me - drop the "also install the Python 2 libs" code and see what happens. I could try to make this change in a couple of test runs in our third-party Cinder CI system and see if something breaks.
snip
Hope this helps,
Sure, thanks! Still, would you agree that for Ussuri this ought to be solved by ripping out the "also install a Python 2 version" part?
G'luck, Peter
On 2019-09-10 01:23:04 +0300 (+0300), Peter Penchev wrote: [...]
would you agree that for Ussuri this ought to be solved by ripping out the "also install a Python 2 version" part?
At the very least, we ought to hide that functionality behind a config option so it's disabled by default.
participants (4)
-
Clark Boylan
-
Jeremy Stanley
-
Peter Penchev
-
Sean Mooney