Open Stack

On Mon, Sep 9, 2019 at 5:40 PM Peter Penchev <pp at storpool.com> wrote:

> On Mon, Sep 9, 2019 at 2:55 PM Sean Mooney <smooney at redhat.com> wrote:
>
>> On Mon, 2019-09-09 at 11:22 +0300, Peter Penchev wrote:
>> > Hi,
>> >
>> > When devstack's `setup_dev_lib` function is invoked and USE_PYTHON3 has
>> > been specified, this function tries to also install the development
>> library
>> > for Python 2.x, I guess just in case some package has not declared
>> proper
>> > Python 3 support or something. It then proceeds to install the Python 3
>> > version of the library and all its dependencies.
>> >
>> > Unfortunately there is a problem with that, and specifically with script
>> > files installed in the system's executable files directory, e.g.
>> > /usr/local/bin. The problem appears when some Python library has already
>> > been installed for Python 3 (and has installed its script files), but is
>> > now installed for Python 2 (overwriting the script files) and is then
>> not
>> > forcefully reinstalled for Python 3, since it is already present. Thus,
>> the
>> > script files are last modified by the Python 2 library installation and
>> > they have a hashbang line saying `python2.x` - so if something then
>> tries
>> > to execute them, they will run and use modules and libraries for Python
>> 2
>> > only.
>> yes this is a long standing issue. we discovered it a year ago but it was
>> never fix.
>>
>> in Ussrui i guess one of the first changes to devstack  to make it python
>> 3 only
>> will be to chagne that behavior. im not sure if we will be able to change
>> it before then.
>>
>> whenever you us libs_from_git in your local.conf on a python 3 install it
>> will install
>> them twice both with python 2 and python 3.  i hope more distros elect to
>> symlink /usr/bin/python to python 3
>> some distros have chosen to do that on systems that are python only and i
>> believe that is the correct
>> approch.
>>
>> when i encountered this it was always resuliting on the script header
>> being #!/usr/bin/python  with no
>> version suffix i
>> gues on a system where that points to python 3 the python 2.7 install
>> might write python2.7
>> there instead?
>>
>
> It depends on what version of pip is invoked; I think that the way
> devstack invokes it nowadays it will always provide a version on the
> shebang line.
>
>
>>
>> >
>> > We experienced this problem when running the cinderlib tests from
>> Cinder's
>> > `playbooks/cinderlib-run.yaml` file - it finds a unit2 executable
>> > (installed by the unittest2 library) and runs it, hoping that unit2
>> will be
>> > able to discover and collect the cinderlib tests and load the cinderlib
>> > modules. However, since unittest2 has last been installed as a Python 2
>> > library, unit2 runs with Python 2 and fails to locate the cinderlib
>> > modules. (Yes, we know that there are other ways to run the cinderlib
>> > tests; this message is about the problem exposed by this way of running
>> > them)
>> >
>> > The obvious solution would be to instruct the Python 2 pip to not
>> install
>> > script (or other shared) files at all; unfortunately,
>> > https://github.com/pypa/pip/issues/3980 ("Option to exclude scripts on
>> > install"), detailing a very similar use case ("need it installed for
>> Python
>> > 2, but want to use it with Python 3") has been open for almost exactly
>> > three years now with no progress. I wonder if I could try to help, but
>> even
>> > if this issue is resolved, there will be some time before OpenStack can
>> > actually depend on a recent enough version of pip.
>> well the obvious solution is to stop doing this entirly.
>> it was added as a hack to ensure if you use  LIB_FROM_GIT in you
>> local.conf that those
>> libs would always be install from the git checkout that you specified in
>> you local.conf
>> for train we are technically requireing all project to run under python 3
>> so we could remove
>> the fallback mechanium of in stalling under python 2. it was there incase
>> a service installed
>> under python 2 to ensure it used the same version of the lib and did not
>> use a version form
>> pypi instead. i wanted to stop doing this last year but we could not
>> becase not all project
>> could run under python 3. but now that they should be able to we dont
>> need this hack anymore.
>> we should change it to respec the python version you have selected. that
>> will speed
>> up stacking speed as we wont have to install everything twice and fix the
>> issue you have encountered.
>>
>
> Yeah, thanks for confirming my thoughts that this might be the right
> solution. I've proposed https://review.opendev.org/681029/ (and set
> workflow -1) to wait for the Ussuri cycle.
>
>
>> >
>> > A horrible workaround would be to find the binary directory before
>> > installing the Python 2 library (using something like `pip3.7 show
>> > somepackage` and then running some heuristics on the "Location" field),
>> > tar'ing it up and then restoring it... but I don't know if I even want
>> to
>> > think about this.
>> >
>> > Another possible way forward would be to consider whether we still want
>> the
>> > Python 2 libraries installed - is OpenStack's Python 3 transition
>> reached a
>> > far enough stage to assume that any projects that still require Python 2
>> > *and* fail to declare their Python 2 dependencies properly are buggy?
>> To be
>> > honest, this seems the most reasonable path for me - drop the "also
>> install
>> > the Python 2 libs" code and see what happens. I could try to make this
>> > change in a couple of test runs in our third-party Cinder CI system and
>> see
>> > if something breaks.
>>
>
> G'luck,
> Peter
>
> Argh, I sent this from the wrong account, did I not...

G'luck,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190909/7aa0acbb/attachment.html>