[dev][designate] PSA: pkg_resources is broken with newer oslo wheels on Python 3.8
o/ A quick PSA for a topic has come up in a few different OpenStack forums of late. As described in a recently filed setuptools bug [1], the pkg_resources found in the most recent version of setuptools to support Python 3.8 does not correctly normalize package names passed to `pkg_resources.require`. This has become an issue of late because since earlier this year the OpenStack release tooling has started (correctly) normalizing filenames of the '.whl' files uploaded to PyPI (so e.g. 'oslo.messaging' now becomes 'oslo_messaging'). You are likely to see this manifest with weird errors like `The 'package.foo>=1.2.3' distribution was not found`, when you know that it is. I have seen it pop up because either (a) pkg_resources is being invoked via an explicit call, as in some of the release tooling used in `openstack/releases`, or (b) pkg_resources is being invoked via the auto-generated import headers added by setuptools when you use its legacy "scripts" keyword rather than newer "console scripts" replacement. The latter in particular is biting some projects indirectly due to Designate and Octavia incorrectly using this mechanism to work around the demise of wsgi_scripts, and I have fixes up waiting for review [2][3]. Just sharing in case some people have seen this, because it's quite a weird corner case to hit. Stephen [1] https://github.com/pypa/setuptools/issues/5051 [2] https://review.opendev.org/c/openstack/designate/+/950569/ [3] https://review.opendev.org/c/openstack/designate/+/955683/
Thank you for clarifying what is going on with the package naming, it has been hard to follow the IRC conversations. As for the proposed patches, they are stuck because when we run a devstack test with mod_wsgi enabled (not my choice by far), it is failing with these patches. I have rechecked to see if the issue was in devstack and has been corrected. I selected the "scripts" approach (which isn't listed as deprecated here[1]) as it was a straightforward "you get the script that you used to get" replacement for the wsgi_scripts entrypoint. Michael [1] https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html On Thu, Jul 24, 2025 at 4:59 AM Stephen Finucane <stephenfin@redhat.com> wrote:
o/
A quick PSA for a topic has come up in a few different OpenStack forums of late.
As described in a recently filed setuptools bug [1], the pkg_resources found in the most recent version of setuptools to support Python 3.8 does not correctly normalize package names passed to `pkg_resources.require`. This has become an issue of late because since earlier this year the OpenStack release tooling has started (correctly) normalizing filenames of the '.whl' files uploaded to PyPI (so e.g. 'oslo.messaging' now becomes 'oslo_messaging').
You are likely to see this manifest with weird errors like `The 'package.foo>=1.2.3' distribution was not found`, when you know that it is. I have seen it pop up because either (a) pkg_resources is being invoked via an explicit call, as in some of the release tooling used in `openstack/releases`, or (b) pkg_resources is being invoked via the auto-generated import headers added by setuptools when you use its legacy "scripts" keyword rather than newer "console scripts" replacement. The latter in particular is biting some projects indirectly due to Designate and Octavia incorrectly using this mechanism to work around the demise of wsgi_scripts, and I have fixes up waiting for review [2][3].
Just sharing in case some people have seen this, because it's quite a weird corner case to hit.
Stephen
[1] https://github.com/pypa/setuptools/issues/5051 [2] https://review.opendev.org/c/openstack/designate/+/950569/ [3] https://review.opendev.org/c/openstack/designate/+/955683/
On Thu, 2025-07-24 at 14:09 -0700, Michael Johnson wrote:
Thank you for clarifying what is going on with the package naming, it has been hard to follow the IRC conversations.
As for the proposed patches, they are stuck because when we run a devstack test with mod_wsgi enabled (not my choice by far), it is failing with these patches. I have rechecked to see if the issue was in devstack and has been corrected.
I've note this a few times on IRC, but the way the patches are structured ensures things are working with uWSGI when used with a WSGI script rather than a module path [1], though I also have follow-up patches to show that module path-based deployments work too [2]. I didn't consider testing with mod_wsgi but it appears that's already broken on at least the stable/2025.1 branch with or without my changes [3]?
I selected the "scripts" approach (which isn't listed as deprecated here[1]) as it was a straightforward "you get the script that you used to get" replacement for the wsgi_scripts entrypoint.
Interesting. That looks like a gap in the setuptools docs. They're listed as discouraged here [4] with more discussion on why here [5]. Unfortunately, as we're now seeing, 'scripts' isn't just a straight forward copy and it modifies the scripts to rewrite imports so it's not a great replacement either. Stephen [1] https://review.opendev.org/c/openstack/designate/+/950569 [2] https://review.opendev.org/c/openstack/designate/+/950633 [1] https://review.opendev.org/c/openstack/designate/+/956070 [4] https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setupto... [5] https://discuss.python.org/t/whats-the-status-of-scripts-vs-entry-points/185...
Michael
[1] https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html
On Thu, Jul 24, 2025 at 4:59 AM Stephen Finucane <stephenfin@redhat.com> wrote:
o/
A quick PSA for a topic has come up in a few different OpenStack forums of late.
As described in a recently filed setuptools bug [1], the pkg_resources found in the most recent version of setuptools to support Python 3.8 does not correctly normalize package names passed to `pkg_resources.require`. This has become an issue of late because since earlier this year the OpenStack release tooling has started (correctly) normalizing filenames of the '.whl' files uploaded to PyPI (so e.g. 'oslo.messaging' now becomes 'oslo_messaging').
You are likely to see this manifest with weird errors like `The 'package.foo>=1.2.3' distribution was not found`, when you know that it is. I have seen it pop up because either (a) pkg_resources is being invoked via an explicit call, as in some of the release tooling used in `openstack/releases`, or (b) pkg_resources is being invoked via the auto-generated import headers added by setuptools when you use its legacy "scripts" keyword rather than newer "console scripts" replacement. The latter in particular is biting some projects indirectly due to Designate and Octavia incorrectly using this mechanism to work around the demise of wsgi_scripts, and I have fixes up waiting for review [2][3].
Just sharing in case some people have seen this, because it's quite a weird corner case to hit.
Stephen
[1] https://github.com/pypa/setuptools/issues/5051 [2] https://review.opendev.org/c/openstack/designate/+/950569/ [3] https://review.opendev.org/c/openstack/designate/+/955683/
Hello o/ Just noticed we’re hitting the same issue in the Gnocchi projects CI for the stable/4.6 branch as that still tests using Python 3.8 and Python 3.9 that seems to get the same issue due to setuptools version. I’m not really sure what can be done, I guess the two alternatives for us there is to pin oslo versions? Or wait for a fix in setuptools that might not arrive (or atleast arrive soon). Open to any ideas :) /Tobias
On 24 Jul 2025, at 13:58, Stephen Finucane <stephenfin@redhat.com> wrote:
o/
A quick PSA for a topic has come up in a few different OpenStack forums of late.
As described in a recently filed setuptools bug [1], the pkg_resources found in the most recent version of setuptools to support Python 3.8 does not correctly normalize package names passed to `pkg_resources.require`. This has become an issue of late because since earlier this year the OpenStack release tooling has started (correctly) normalizing filenames of the '.whl' files uploaded to PyPI (so e.g. 'oslo.messaging' now becomes 'oslo_messaging').
You are likely to see this manifest with weird errors like `The 'package.foo>=1.2.3' distribution was not found`, when you know that it is. I have seen it pop up because either (a) pkg_resources is being invoked via an explicit call, as in some of the release tooling used in `openstack/releases`, or (b) pkg_resources is being invoked via the auto-generated import headers added by setuptools when you use its legacy "scripts" keyword rather than newer "console scripts" replacement. The latter in particular is biting some projects indirectly due to Designate and Octavia incorrectly using this mechanism to work around the demise of wsgi_scripts, and I have fixes up waiting for review [2][3].
Just sharing in case some people have seen this, because it's quite a weird corner case to hit.
Stephen
[1] https://github.com/pypa/setuptools/issues/5051 [2] https://review.opendev.org/c/openstack/designate/+/950569/ [3] https://review.opendev.org/c/openstack/designate/+/955683/
participants (3)
-
Michael Johnson
-
Stephen Finucane
-
Tobias Urdin - Binero