On Mon, Aug 31, 2020 at 4:32 AM Ian Wienand <iwienand@redhat.com> wrote:
On Fri, Jul 03, 2020 at 12:13:04PM -0700, Clark Boylan wrote:
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
This is now back with setuptools 50.0.0 [1], see the original issue [2].
The problems are limited to instances where jobs are installing with pip as root into the system environment on platforms that override the default install path (debuntu). The confluence of this set of requirements of neatly describes most devstack testing :/
There's two visible problems; both stem from the same issue. Packaged Debuntu python installs things into dist-packages; leaving site-packages for a non-packaged interpreter, should you wish to install such a thing. It patches distutils to provide this behaviour. The other thing it does is makes pip installs use /usr/local/bin, rather that /usr/bin.
Thus it is unfortunately not just s,/usr/local/bin,/usr/bin,g because the new setuptools will install all the libraries into site-packages; which the packaged python intpreter doesn't know to look for.
Using SETUPTOOLS_USE_DISTUTILS=stdlib with such installs is one option; it feels like it just makes for more confusing bifurcation. We can't really set this in stack.sh as a global, because we wouldn't want this to apply to subshells that are installing in virtualenv's, for example. It might be the best option.
A more radical thought; perhaps we could install a non-packaged python interpreter for devstack runs. Isolation from packaging cuts both ways; while we might work around packaging issues in CI, we're also working around packaging issues that then just hit you when you're in production. The eternal question of "what are we testing".
I don't think there's an easy answer. Which is probably why we've ended up here with everything broken ...
Is it the right time to discuss switching to virtual environments? We've had quite a positive experience with bifrost since we stopped trying global installations at all. Dmitry
-i
[1] https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c4... [2] https://github.com/pypa/setuptools/issues/2232