Open Stack

On Mon, Sep 9, 2019 at 7:42 PM Clark Boylan <cboylan at sapwetik.org> wrote:

> On Mon, Sep 9, 2019, at 1:22 AM, Peter Penchev wrote:
> > Hi,
> >
> > When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has
> > been specified, this function tries to also install the development
> > library for Python 2.x, I guess just in case some package has not
> > declared proper Python 3 support or something. It then proceeds to
> > install the Python 3 version of the library and all its dependencies.
> >
> > Unfortunately there is a problem with that, and specifically with
> > script files installed in the system's executable files directory, e.g.
> > /usr/local/bin. The problem appears when some Python library has
> > already been installed for Python 3 (and has installed its script
> > files), but is now installed for Python 2 (overwriting the script
> > files) and is then not forcefully reinstalled for Python 3, since it is
> > already present. Thus, the script files are last modified by the Python
> > 2 library installation and they have a hashbang line saying `python2.x`
> > - so if something then tries to execute them, they will run and use
> > modules and libraries for Python 2 only.
> >
> > We experienced this problem when running the cinderlib tests from
> > Cinder's `playbooks/cinderlib-run.yaml` file - it finds a unit2
> > executable (installed by the unittest2 library) and runs it, hoping
> > that unit2 will be able to discover and collect the cinderlib tests and
> > load the cinderlib modules. However, since unittest2 has last been
> > installed as a Python 2 library, unit2 runs with Python 2 and fails to
> > locate the cinderlib modules. (Yes, we know that there are other ways
> > to run the cinderlib tests; this message is about the problem exposed
> > by this way of running them)
>
> One option here is to explicitly run the file under the python version you
> want. I do this with `pbr freeze` frequently to ensure I'm looking at the
> correct version of software for the correct version of python. For example:
>
>   python3 /usr/local/bin/pbr freeze | grep $packagename
>   python2 /usr/local/bin/pbr freeze | grep $packagename
>
> Then as long as you have installed the utility (in my case pbr) under both
> python versions it should just work assuming they don't write different
> files for different versions of python at install time.
>

This is what we ended up doing (sorry, I might have mentioned that in the
original message; it was a solved problem for our CI) - we modified the
Ansible job to explicitly run "python3.7 unit2". So, yeah, my message was
more to point out the general problem than to ask for help for our specific
case, but still, yeah, thanks, that's exactly what we did.

>
> > The obvious solution would be to instruct the Python 2 pip to not
> > install script (or other shared) files at all; unfortunately,
> > https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on
> > install"), detailing a very similar use case ("need it installed for
> > Python 2, but want to use it with Python 3") has been open for almost
> > exactly three years now with no progress. I wonder if I could try to
> > help, but even if this issue is resolved, there will be some time
> > before OpenStack can actually depend on a recent enough version of pip.
>
> Note OpenStack tests with, and as a result possibly requires, the latest
> version of pip. Fixing this in pip shouldn't be a problem as long as they
> make a release not long after.
>

Right, I did briefly wonder whether this was true while writing my mail, I
should have taken the time to check and see that devstack actually installs
its own version of pip and removes any versions installed by OS packages.
Hm, I just might try my hand at that in the coming days or weeks, but I
can't really make any promises.


> >
> > A horrible workaround would be to find the binary directory before
> > installing the Python 2 library (using something like `pip3.7 show
> > somepackage` and then running some heuristics on the "Location" field),
> > tar'ing it up and then restoring it... but I don't know if I even want
> > to think about this.
> >
> > Another possible way forward would be to consider whether we still want
> > the Python 2 libraries installed - is OpenStack's Python 3 transition
> > reached a far enough stage to assume that any projects that still
> > require Python 2 *and* fail to declare their Python 2 dependencies
> > properly are buggy? To be honest, this seems the most reasonable path
> > for me - drop the "also install the Python 2 libs" code and see what
> > happens. I could try to make this change in a couple of test runs in
> > our third-party Cinder CI system and see if something breaks.
> >
>
> snip
>
> Hope this helps,
>

Sure, thanks! Still, would you agree that for Ussuri this ought to be
solved by ripping out the "also install a Python 2 version" part?

G'luck,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190910/65f17486/attachment-0001.html>