Open Stack

On Fri, Sep 27, 2019 at 11:09:22AM +0000, Jeremy Stanley wrote:
> I'd eventually love to see us stop preinstalling pip and virtualenv
> entirely, allowing jobs to take care of doing that at runtime if
> they need to use them.

You'd think, right?  :) But it is a bit of a can of worms ...

So pip is a part of Python 3 ... "dnf install python3" brings in
python3-pip unconditionally.  So there will always be a pip on the
host.

For CentOS 8 that's pip version 9.something (upstream is on
19.something).  This is where traditionally we've had problems;
requirements, etc. uses some syntax feature that tickles a bug in old
pip and we're back to trying to override the default version.  I think
we can agree to try and mitigate that in jobs, rather than in base
images.

But as an additional complication, CentOS 8 ships it's
"platform-python" which is used by internal tools like dnf.  The thing
is, we have Python tools that could probably reasonably be considered
platform tools like "glean" which instantiates our networking.  I am
not sure if "/usr/libexec/platform-python -m pip install glean" is
considered an abuse or a good way to install against a stable Python
version.  I'll go with the latter ...

But ... platform-python doesn't have virtualenv (separate package on
Python 3).  Python documentation says that "venv" is a good way to
create a virtual environment and basically suggests it can do things
better than virtualenv because it's part of the base Python and so
doesn't have to have a bunch of hacks.  Then the virtualenv
documentation throws some shade at venv saying "a subset of
[virtualenv] has been integrated into the standard library" and lists
why virtualenv is better.  Now we have *three* choices for a virtual
environment: venv with either platform python or packaged python, or
virtualenv with packaged python.  Which should an element choose, if
they want to setup tools on the host during image build?  And how do
we stop every element having to hard-code all this logic into itself
over and over?

Where I came down on this is :

https://review.opendev.org/684462 : this stops installing from source
on CentOS 8, which I think we all agree on.  It makes some opinionated
decisions in creating DIB_PYTHON_PIP and DIB_PYTHON_VIRTUALENV
variables that will "do the right thing" when used by elements:

 * Python 2 first era (trusty/centos7) will use python2 pip and virtualenv
 * Python 3 era (bionic/fedora) will use python3 pip and venv (*not*
   virtualenv)
 * RHEL8/CentOS 8 will use platform-python pip & venv

https://review.opendev.org/685643 : above in action; installing glean
correctly on all supported distros.

-i