On Fri, Jul 03, 2020 at 12:13:04PM -0700, Clark Boylan wrote:
> Setuptools has made a new version 48 release. This appears to be
> causing problems for devstack because `pip install -e $PACKAGE_PATH`
> installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it
> did in the past. `pip install $PACKAGE_PATH` continues to install to
> /usr/local/bin as expected. Devstack is failing because
> keystone-manage cannot currently be found at the specific
> /usr/local/bin/ path.

This is now back with setuptools 50.0.0 [1], see the original issue
[2].

The problems are limited to instances where jobs are installing with
pip as root into the system environment on platforms that override the
default install path (debuntu).  The confluence of this set of
requirements of neatly describes most devstack testing :/

There's two visible problems; both stem from the same issue.  Packaged
Debuntu python installs things into dist-packages; leaving
site-packages for a non-packaged interpreter, should you wish to
install such a thing.  It patches distutils to provide this behaviour.
The other thing it does is makes pip installs use /usr/local/bin,
rather that /usr/bin.

Thus it is unfortunately not just s,/usr/local/bin,/usr/bin,g because
the new setuptools will install all the libraries into site-packages;
which the packaged python intpreter doesn't know to look for.

Using SETUPTOOLS_USE_DISTUTILS=stdlib with such installs is one
option; it feels like it just makes for more confusing bifurcation.
We can't really set this in stack.sh as a global, because we wouldn't
want this to apply to subshells that are installing in virtualenv's,
for example.  It might be the best option.

A more radical thought; perhaps we could install a non-packaged python
interpreter for devstack runs.  Isolation from packaging cuts both
ways; while we might work around packaging issues in CI, we're also
working around packaging issues that then just hit you when you're in
production.  The eternal question of "what are we testing".

I don't think there's an easy answer.  Which is probably why we've
ended up here with everything broken ...

-i

[1] https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c44b7c85
[2] https://github.com/pypa/setuptools/issues/2232