Setuptools 48 and Devstack Failures
Hello, Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path. Potential workarounds for this include not using `pip install -e` or relying on $PATH to find the commands rather than specifying rooted paths to them. I'll defer to the QA team on how they want to address this. While we can have devstack install an older setuptools version as well, generally this is not considered to be a good idea because anyone doing pip installs outside of devstack may get the newer behavior. It is actually important for us to try and keep up with setuptools changes as a result. Fungi indicated that setuptools expected this to be a bumpy upgrade. I'm not sure if they would consider `pip install -e` and `pip install` installing to different paths as a bug, and if they did which behavior is correct. It would probably be a good idea to file a bug upstream if we debug this further. Clark
---- On Fri, 03 Jul 2020 14:13:04 -0500 Clark Boylan <cboylan@sapwetik.org> wrote ----
Hello,
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
Potential workarounds for this include not using `pip install -e` or relying on $PATH to find the commands rather than specifying rooted paths to them. I'll defer to the QA team on how they want to address this. While we can have devstack install an older setuptools version as well, generally this is not considered to be a good idea because anyone doing pip installs outside of devstack may get the newer behavior. It is actually important for us to try and keep up with setuptools changes as a result.
Fungi indicated that setuptools expected this to be a bumpy upgrade. I'm not sure if they would consider `pip install -e` and `pip install` installing to different paths as a bug, and if they did which behavior is correct. It would probably be a good idea to file a bug upstream if we debug this further.
Yeah, I am not sure how it will go as setuptools bug or an incompatible change and needs to handle on devstack side. As this is blocking all gates, let's use the old setuptools temporarily. For now, I filed devstack bug to track it and once we figure it out then move to latest setuptools - https://bugs.launchpad.net/devstack/+bug/1886237 This is patch to use old setuptools- - https://review.opendev.org/#/c/739290/
Clark
---- On Fri, 03 Jul 2020 17:29:18 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Fri, 03 Jul 2020 14:13:04 -0500 Clark Boylan <cboylan@sapwetik.org> wrote ----
Hello,
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
Potential workarounds for this include not using `pip install -e` or relying on $PATH to find the commands rather than specifying rooted paths to them. I'll defer to the QA team on how they want to address this. While we can have devstack install an older setuptools version as well, generally this is not considered to be a good idea because anyone doing pip installs outside of devstack may get the newer behavior. It is actually important for us to try and keep up with setuptools changes as a result.
Fungi indicated that setuptools expected this to be a bumpy upgrade. I'm not sure if they would consider `pip install -e` and `pip install` installing to different paths as a bug, and if they did which behavior is correct. It would probably be a good idea to file a bug upstream if we debug this further.
Yeah, I am not sure how it will go as setuptools bug or an incompatible change and needs to handle on devstack side. As this is blocking all gates, let's use the old setuptools temporarily. For now, I filed devstack bug to track it and once we figure it out then move to latest setuptools - https://bugs.launchpad.net/devstack/+bug/1886237
This is patch to use old setuptools- - https://review.opendev.org/#/c/739290/
Updates: Issue is when setuptools adopts distutils from the standard library (in 48.0.0) and uses it, downstream packagers customization to distutils will be lost. - https://github.com/pypa/setuptools/issues/2232 setuptools 49.1.0 reverted the adoption of distutils from the standard library and its working now. I have closed the devstack bug 1886237 and proposed the revert of capping of setuptools by blacklisting 48.0.0 and 49.0.0 so that we test with latest setuptools. For now, devstack will pick the 49.1.0 and pass. - https://review.opendev.org/#/c/739294/2 In summary, gate is green and you can recheck on the failed patches. -gmann
Clark
On 20-07-04 20:24:55, Ghanshyam Mann wrote:
---- On Fri, 03 Jul 2020 17:29:18 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Fri, 03 Jul 2020 14:13:04 -0500 Clark Boylan <cboylan@sapwetik.org> wrote ----
Hello,
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
Potential workarounds for this include not using `pip install -e` or relying on $PATH to find the commands rather than specifying rooted paths to them. I'll defer to the QA team on how they want to address this. While we can have devstack install an older setuptools version as well, generally this is not considered to be a good idea because anyone doing pip installs outside of devstack may get the newer behavior. It is actually important for us to try and keep up with setuptools changes as a result.
Fungi indicated that setuptools expected this to be a bumpy upgrade. I'm not sure if they would consider `pip install -e` and `pip install` installing to different paths as a bug, and if they did which behavior is correct. It would probably be a good idea to file a bug upstream if we debug this further.
Yeah, I am not sure how it will go as setuptools bug or an incompatible change and needs to handle on devstack side. As this is blocking all gates, let's use the old setuptools temporarily. For now, I filed devstack bug to track it and once we figure it out then move to latest setuptools - https://bugs.launchpad.net/devstack/+bug/1886237
This is patch to use old setuptools- - https://review.opendev.org/#/c/739290/
Updates: Issue is when setuptools adopts distutils from the standard library (in 48.0.0) and uses it, downstream packagers customization to distutils will be lost. - https://github.com/pypa/setuptools/issues/2232
setuptools 49.1.0 reverted the adoption of distutils from the standard library and its working now.
I have closed the devstack bug 1886237 and proposed the revert of capping of setuptools by blacklisting 48.0.0 and 49.0.0 so that we test with latest setuptools. For now, devstack will pick the 49.1.0 and pass. - https://review.opendev.org/#/c/739294/2
In summary, gate is green and you can recheck on the failed patches.
It looks like they (upstream) are rolling forward with the change. There are workarounds for those that need it (env var). Please see the above linked issue for more information.
-gmann
Clark
-- Matthew Thode
On Fri, Jul 03, 2020 at 12:13:04PM -0700, Clark Boylan wrote:
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
This is now back with setuptools 50.0.0 [1], see the original issue [2]. The problems are limited to instances where jobs are installing with pip as root into the system environment on platforms that override the default install path (debuntu). The confluence of this set of requirements of neatly describes most devstack testing :/ There's two visible problems; both stem from the same issue. Packaged Debuntu python installs things into dist-packages; leaving site-packages for a non-packaged interpreter, should you wish to install such a thing. It patches distutils to provide this behaviour. The other thing it does is makes pip installs use /usr/local/bin, rather that /usr/bin. Thus it is unfortunately not just s,/usr/local/bin,/usr/bin,g because the new setuptools will install all the libraries into site-packages; which the packaged python intpreter doesn't know to look for. Using SETUPTOOLS_USE_DISTUTILS=stdlib with such installs is one option; it feels like it just makes for more confusing bifurcation. We can't really set this in stack.sh as a global, because we wouldn't want this to apply to subshells that are installing in virtualenv's, for example. It might be the best option. A more radical thought; perhaps we could install a non-packaged python interpreter for devstack runs. Isolation from packaging cuts both ways; while we might work around packaging issues in CI, we're also working around packaging issues that then just hit you when you're in production. The eternal question of "what are we testing". I don't think there's an easy answer. Which is probably why we've ended up here with everything broken ... -i [1] https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c4... [2] https://github.com/pypa/setuptools/issues/2232
On Mon, Aug 31, 2020 at 12:27:33PM +1000, Ian Wienand wrote:
Thus it is unfortunately not just s,/usr/local/bin,/usr/bin,g because the new setuptools will install all the libraries into site-packages; which the packaged python intpreter doesn't know to look for.
https://review.opendev.org/748937 was where I tried this before I understood the above.
Using SETUPTOOLS_USE_DISTUTILS=stdlib with such installs is one option; it feels like it just makes for more confusing bifurcation. We can't really set this in stack.sh as a global, because we wouldn't want this to apply to subshells that are installing in virtualenv's, for example. It might be the best option.
https://review.opendev.org/748957 is this option; this should hook into pip_install function. How many plugins do "sudo pip install ..." I don't know; they would all still be broken with this. But as mentioned, we don't want to set this globally to avoid setting it for virtualenv installs.
A more radical thought; perhaps we could install a non-packaged python interpreter for devstack runs. Isolation from packaging cuts both ways; while we might work around packaging issues in CI, we're also working around packaging issues that then just hit you when you're in production. The eternal question of "what are we testing".
On further consideration I don't think this is a great idea. Lots of things do #!/usr/bin/python3 which is always going to be the packaged Python. I imagine we'd have quite a mess of things not understanding which python their libraries are installed for. Another thing that failed was just using the system packaged pip; https://review.opendev.org/748942. In theory that would be OK, and obviously patched correctly for the distro, but unfortunately the bionic pip is so old it doesn't pull down manylinux2010 wheels and so there's assorted build breakages from packages that now have to build. https://review.opendev.org/748943/ is a pin to <50 in requirements. devstack uses requirements to install setuptools in it's tools/install_pip.sh so this does move the system back to a version without this change. Obviously this doesn't fix the underlying problem, but helps the gate. -i
On Mon, Aug 31, 2020 at 4:32 AM Ian Wienand <iwienand@redhat.com> wrote:
On Fri, Jul 03, 2020 at 12:13:04PM -0700, Clark Boylan wrote:
Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path.
This is now back with setuptools 50.0.0 [1], see the original issue [2].
The problems are limited to instances where jobs are installing with pip as root into the system environment on platforms that override the default install path (debuntu). The confluence of this set of requirements of neatly describes most devstack testing :/
There's two visible problems; both stem from the same issue. Packaged Debuntu python installs things into dist-packages; leaving site-packages for a non-packaged interpreter, should you wish to install such a thing. It patches distutils to provide this behaviour. The other thing it does is makes pip installs use /usr/local/bin, rather that /usr/bin.
Thus it is unfortunately not just s,/usr/local/bin,/usr/bin,g because the new setuptools will install all the libraries into site-packages; which the packaged python intpreter doesn't know to look for.
Using SETUPTOOLS_USE_DISTUTILS=stdlib with such installs is one option; it feels like it just makes for more confusing bifurcation. We can't really set this in stack.sh as a global, because we wouldn't want this to apply to subshells that are installing in virtualenv's, for example. It might be the best option.
A more radical thought; perhaps we could install a non-packaged python interpreter for devstack runs. Isolation from packaging cuts both ways; while we might work around packaging issues in CI, we're also working around packaging issues that then just hit you when you're in production. The eternal question of "what are we testing".
I don't think there's an easy answer. Which is probably why we've ended up here with everything broken ...
Is it the right time to discuss switching to virtual environments? We've had quite a positive experience with bifrost since we stopped trying global installations at all. Dmitry
-i
[1] https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c4... [2] https://github.com/pypa/setuptools/issues/2232
participants (5)
-
Clark Boylan
-
Dmitry Tantsur
-
Ghanshyam Mann
-
Ian Wienand
-
Matthew Thode