[ci][kolla] setuptools v82.0.0 breaks a bunch of things by removing pkg_resources
Hey all, I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release. setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? Michael
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment that it was run in most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it.
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib.... i do not see kolla in https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos= what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Michael
I've not checked the exact problem in kolla, but an example of the problems triggered by this is failure during installing bunch of xstatic-* libraries for horizon --- 2026-02-09 06:18:16.431558 | controller | × Getting requirements to build wheel did not run successfully. 2026-02-09 06:18:16.431562 | controller | │ exit code: 1 2026-02-09 06:18:16.431576 | controller | ╰─> [23 lines of output] 2026-02-09 06:18:16.431580 | controller | Traceback (most recent call last): 2026-02-09 06:18:16.431589 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> 2026-02-09 06:18:16.431594 | controller | main() 2026-02-09 06:18:16.431606 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main 2026-02-09 06:18:16.431610 | controller | json_out["return_val"] = hook(**hook_input["kwargs"]) 2026-02-09 06:18:16.431623 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431628 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel 2026-02-09 06:18:16.431640 | controller | return hook(config_settings) 2026-02-09 06:18:16.431644 | controller | ^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431648 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel 2026-02-09 06:18:16.431661 | controller | return self._get_build_requires(config_settings, requirements=[]) 2026-02-09 06:18:16.431668 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431672 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires 2026-02-09 06:18:16.431676 | controller | self.run_setup() 2026-02-09 06:18:16.431689 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 520, in run_setup 2026-02-09 06:18:16.431693 | controller | super().run_setup(setup_script=setup_script) 2026-02-09 06:18:16.431697 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup 2026-02-09 06:18:16.431710 | controller | exec(code, locals()) 2026-02-09 06:18:16.431714 | controller | File "<string>", line 1, in <module> 2026-02-09 06:18:16.431727 | controller | File "/tmp/pip-install-2pd_mk_v/xstatic-angular-schema-form_a23b90a319e14f1fb57c9d4525797f23/xstatic/__init__.py", line 1, in <module> 2026-02-09 06:18:16.431731 | controller | __import__('pkg_resources').declare_namespace(__name__) 2026-02-09 06:18:16.431735 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431747 | controller | ModuleNotFoundError: No module named 'pkg_resources' 2026-02-09 06:18:16.431751 | controller | [end of output] 2026-02-09 06:18:16.431755 | controller | --- The tricky part is that it uses pkg_resources to declare namespace packages, which can't be directly replaced by importlib. In the past I attempted to re-implement it using namespace packages explained in the below article but didn't get it working correctly... (due to interaction with pbr, as far as I remember). https://packaging.python.org/en/latest/guides/packaging-namespace-packages/ On 2/9/26 7:35 PM, Sean Mooney wrote:
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment that it was run in most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it.
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib....
i do not see kolla in https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos=
what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Michael
Hi, I’ve raised a horizon bug - https://bugs.launchpad.net/horizon/+bug/2141293 and a kolla setuptools pin patch for now https://review.opendev.org/c/openstack/kolla/+/976099. Best regards, Bartosz Bezak
On 9 Feb 2026, at 11:43, Takashi Kajinami <kajinamit@oss.nttdata.com> wrote:
I've not checked the exact problem in kolla, but an example of the problems triggered by this is failure during installing bunch of xstatic-* libraries for horizon
--- 2026-02-09 06:18:16.431558 | controller | × Getting requirements to build wheel did not run successfully. 2026-02-09 06:18:16.431562 | controller | │ exit code: 1 2026-02-09 06:18:16.431576 | controller | ╰─> [23 lines of output] 2026-02-09 06:18:16.431580 | controller | Traceback (most recent call last): 2026-02-09 06:18:16.431589 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> 2026-02-09 06:18:16.431594 | controller | main() 2026-02-09 06:18:16.431606 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main 2026-02-09 06:18:16.431610 | controller | json_out["return_val"] = hook(**hook_input["kwargs"]) 2026-02-09 06:18:16.431623 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431628 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel 2026-02-09 06:18:16.431640 | controller | return hook(config_settings) 2026-02-09 06:18:16.431644 | controller | ^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431648 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel 2026-02-09 06:18:16.431661 | controller | return self._get_build_requires(config_settings, requirements=[]) 2026-02-09 06:18:16.431668 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431672 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires 2026-02-09 06:18:16.431676 | controller | self.run_setup() 2026-02-09 06:18:16.431689 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 520, in run_setup 2026-02-09 06:18:16.431693 | controller | super().run_setup(setup_script=setup_script) 2026-02-09 06:18:16.431697 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup 2026-02-09 06:18:16.431710 | controller | exec(code, locals()) 2026-02-09 06:18:16.431714 | controller | File "<string>", line 1, in <module> 2026-02-09 06:18:16.431727 | controller | File "/tmp/pip-install-2pd_mk_v/xstatic-angular-schema-form_a23b90a319e14f1fb57c9d4525797f23/xstatic/__init__.py", line 1, in <module> 2026-02-09 06:18:16.431731 | controller | __import__('pkg_resources').declare_namespace(__name__) 2026-02-09 06:18:16.431735 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431747 | controller | ModuleNotFoundError: No module named 'pkg_resources' 2026-02-09 06:18:16.431751 | controller | [end of output] 2026-02-09 06:18:16.431755 | controller | ---
The tricky part is that it uses pkg_resources to declare namespace packages, which can't be directly replaced by importlib.
In the past I attempted to re-implement it using namespace packages explained in the below article but didn't get it working correctly... (due to interaction with pbr, as far as I remember).
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
On 2/9/26 7:35 PM, Sean Mooney wrote:
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment that it was run in most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it. in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib.... i do not see kolla in https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos= what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Michael
On 09/02/2026 10:43, Takashi Kajinami wrote:
I've not checked the exact problem in kolla, but an example of the problems triggered by this is failure during installing bunch of xstatic-* libraries for horizon
yep we just noteiced that in rdo it looks liek the xstatic code has __init__.py files with |__import__('pkg_resources').declare_namespace(__name__) | https://opendev.org/openstack/xstatic-angular-bootstrap/src/branch/master/xs...
--- 2026-02-09 06:18:16.431558 | controller | × Getting requirements to build wheel did not run successfully. 2026-02-09 06:18:16.431562 | controller | │ exit code: 1 2026-02-09 06:18:16.431576 | controller | ╰─> [23 lines of output] 2026-02-09 06:18:16.431580 | controller | Traceback (most recent call last): 2026-02-09 06:18:16.431589 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> 2026-02-09 06:18:16.431594 | controller | main() 2026-02-09 06:18:16.431606 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main 2026-02-09 06:18:16.431610 | controller | json_out["return_val"] = hook(**hook_input["kwargs"]) 2026-02-09 06:18:16.431623 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431628 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel 2026-02-09 06:18:16.431640 | controller | return hook(config_settings) 2026-02-09 06:18:16.431644 | controller | ^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431648 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel 2026-02-09 06:18:16.431661 | controller | return self._get_build_requires(config_settings, requirements=[]) 2026-02-09 06:18:16.431668 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431672 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires 2026-02-09 06:18:16.431676 | controller | self.run_setup() 2026-02-09 06:18:16.431689 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 520, in run_setup 2026-02-09 06:18:16.431693 | controller | super().run_setup(setup_script=setup_script) 2026-02-09 06:18:16.431697 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup 2026-02-09 06:18:16.431710 | controller | exec(code, locals()) 2026-02-09 06:18:16.431714 | controller | File "<string>", line 1, in <module> 2026-02-09 06:18:16.431727 | controller | File "/tmp/pip-install-2pd_mk_v/xstatic-angular-schema-form_a23b90a319e14f1fb57c9d4525797f23/xstatic/__init__.py", line 1, in <module> 2026-02-09 06:18:16.431731 | controller | __import__('pkg_resources').declare_namespace(__name__) 2026-02-09 06:18:16.431735 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431747 | controller | ModuleNotFoundError: No module named 'pkg_resources' 2026-02-09 06:18:16.431751 | controller | [end of output] 2026-02-09 06:18:16.431755 | controller | ---
The tricky part is that it uses pkg_resources to declare namespace packages, which can't be directly replaced by importlib.
In the past I attempted to re-implement it using namespace packages explained in the below article but didn't get it working correctly... (due to interaction with pbr, as far as I remember).
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
i feel like most of these repos have a single package. is this begin done so that all the different xstatic-* packages get merged into one namespace or something like that? i do not understand namespace packages but im wonder if it need to be reimplemente and if just replacying the __init__.py with an empty file would be better. basicly treat this like oslo were each package is actully indepentned with a common prefix rather then living in a single oslo namespace.
On 2/9/26 7:35 PM, Sean Mooney wrote:
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment that it was run in most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it.
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib....
i do not see kolla in https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos=
what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Michael
I think that in case of xstatic, we should go ahead with the xstatic 2.0 plan, which gets rids of the pkg_resources import and also fixes some other problems. The trick is that it's not backward compatible with the current xstatic versions, and the way global requirements currently works, that creates a circular dependency that can't be resolved. On Mon, Feb 9, 2026 at 11:44 AM Takashi Kajinami <kajinamit@oss.nttdata.com> wrote:
I've not checked the exact problem in kolla, but an example of the problems triggered by this is failure during installing bunch of xstatic-* libraries for horizon
--- 2026-02-09 06:18:16.431558 | controller | × Getting requirements to build wheel did not run successfully. 2026-02-09 06:18:16.431562 | controller | │ exit code: 1 2026-02-09 06:18:16.431576 | controller | ╰─> [23 lines of output] 2026-02-09 06:18:16.431580 | controller | Traceback (most recent call last): 2026-02-09 06:18:16.431589 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> 2026-02-09 06:18:16.431594 | controller | main() 2026-02-09 06:18:16.431606 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main 2026-02-09 06:18:16.431610 | controller | json_out["return_val"] = hook(**hook_input["kwargs"]) 2026-02-09 06:18:16.431623 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431628 | controller | File "/opt/stack/data/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel 2026-02-09 06:18:16.431640 | controller | return hook(config_settings) 2026-02-09 06:18:16.431644 | controller | ^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431648 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel 2026-02-09 06:18:16.431661 | controller | return self._get_build_requires(config_settings, requirements=[]) 2026-02-09 06:18:16.431668 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431672 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires 2026-02-09 06:18:16.431676 | controller | self.run_setup() 2026-02-09 06:18:16.431689 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 520, in run_setup 2026-02-09 06:18:16.431693 | controller | super().run_setup(setup_script=setup_script) 2026-02-09 06:18:16.431697 | controller | File "/tmp/pip-build-env-_mvl8qsh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup 2026-02-09 06:18:16.431710 | controller | exec(code, locals()) 2026-02-09 06:18:16.431714 | controller | File "<string>", line 1, in <module> 2026-02-09 06:18:16.431727 | controller | File "/tmp/pip-install-2pd_mk_v/xstatic-angular-schema-form_a23b90a319e14f1fb57c9d4525797f23/xstatic/__init__.py", line 1, in <module> 2026-02-09 06:18:16.431731 | controller | __import__('pkg_resources').declare_namespace(__name__) 2026-02-09 06:18:16.431735 | controller | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-02-09 06:18:16.431747 | controller | ModuleNotFoundError: No module named 'pkg_resources' 2026-02-09 06:18:16.431751 | controller | [end of output] 2026-02-09 06:18:16.431755 | controller | ---
The tricky part is that it uses pkg_resources to declare namespace packages, which can't be directly replaced by importlib.
In the past I attempted to re-implement it using namespace packages explained in the below article but didn't get it working correctly... (due to interaction with pbr, as far as I remember).
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
On 2/9/26 7:35 PM, Sean Mooney wrote:
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks
setuptools doesn't seem to be listed in upper-constraints, perhaps
because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment
a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release. that it was run in
most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it.
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib....
i do not see kolla in https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos=
what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Michael
-- Radomir Dopieralski
Hi, On Mon, Feb 9, 2026 at 4:06 PM Sean Mooney <smooney@redhat.com> wrote:
On 09/02/2026 07:49, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package? for ordering reasons setuptools cant be listed in upper-constraitns historically we have used pip and setup tools to mange the creation of our virutal environemtn an dtox inherited setup tools form the envionment that it was run in most project have adapated to the pkg_resources removal as we were expecting it to happen in october last year but this has been deprecated for removal for 2+ years so we tried to get ahead of it.
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and
https://docs.python.org/3/library/importlib.resources.html#module-importlib....
i do not see kolla in
https://codesearch.opendev.org/?q=pkg_resources&i=nope&literal=nope&files=&excludeFiles=&repos=
what part of the ci broke? was it the stabel jobs? codeseach is just looking at master.
Yes can see many failures in stable/2025.1[1], 2025.2[2] and grenade jobs[3] in master [1] https://zuul.openstack.org/buildset/2983a1701f1245fba7bd5a3b86cf8887 [2] https://zuul.openstack.org/buildset/9c50791d5187426ba9d6b04aab065cc0 [3] https://zuul.openstack.org/build/44fef020d59b49fea0c98ca7465e26fa
Michael
-- Thanks and Regards Yatin Karel
On 2026-02-09 10:35:50 +0000 (+0000), Sean Mooney wrote: [...]
in many cases the replacement is to use https://docs.python.org/3/library/importlib.metadata.html and https://docs.python.org/3/library/importlib.resources.html#module-importlib.... [...]
Fresh from The Better Late Than Never Department, per https://discuss.python.org/t/106079 and today's sudden renewed interest in https://github.com/pypa/setuptools/issues/863 there will likely soon be a separate pkg_resources uploaded to PyPI, which we could make an exception to add to requirements.txt in affected stable branches of projects where heavier overhaul is unwise. -- Jeremy Stanley
On Mon, 2026-02-09 at 18:49 +1100, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package?
Michael
We're seeing this on projects outside of opendev also, such as ORC [1]. The vast majority (though not all, as Takashi notes) of these are due to use of old versions of pbr. These issues are all resolved in recent pbr releases [2]. While we could bump the pbr constraint in upper- constraints for these stable branches, I think it would be wiser to pin setuptools as part of the devstack global venv creation since there were more users of pkg_resources that have only been removed in recent releases. I've proposed devstack changes to do just this for the affected stable branches, which you can find here [3]. Cheers, Stephen [1] https://github.com/k-orc/openstack-resource-controller/actions/runs/21818364... [2] https://review.opendev.org/q/project:openstack/pbr+topic:pep517 [3] https://review.opendev.org/q/topic:%22pin-setuptools%22+owner:stephenfin@red...
Openstack Docs TOX job is broken too: https://zuul.opendev.org/t/openstack/build/e6668bca1a784339a862b31cfdb6aa32 On Mon, Feb 9, 2026 at 2:51 PM Stephen Finucane <stephenfin@redhat.com> wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing
On Mon, 2026-02-09 at 18:49 +1100, Michael Still wrote: pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps
because it's often provided as a distro package?
Michael
We're seeing this on projects outside of opendev also, such as ORC [1]. The vast majority (though not all, as Takashi notes) of these are due to use of old versions of pbr. These issues are all resolved in recent pbr releases [2]. While we could bump the pbr constraint in upper- constraints for these stable branches, I think it would be wiser to pin setuptools as part of the devstack global venv creation since there were more users of pkg_resources that have only been removed in recent releases. I've proposed devstack changes to do just this for the affected stable branches, which you can find here [3].
Cheers, Stephen
[1] https://github.com/k-orc/openstack-resource-controller/actions/runs/21818364... [2] https://review.opendev.org/q/project:openstack/pbr+topic:pep517 [3] https://review.opendev.org/q/topic:%22pin-setuptools%22+owner:stephenfin@red...
-- Regards, Maksim Malchuk
On Mon, 2026-02-09 at 11:50 +0000, Stephen Finucane wrote:
On Mon, 2026-02-09 at 18:49 +1100, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package?
Michael
We're seeing this on projects outside of opendev also, such as ORC [1]. The vast majority (though not all, as Takashi notes) of these are due to use of old versions of pbr. These issues are all resolved in recent pbr releases [2]. While we could bump the pbr constraint in upper- constraints for these stable branches, I think it would be wiser to pin setuptools as part of the devstack global venv creation since there were more users of pkg_resources that have only been removed in recent releases. I've proposed devstack changes to do just this for the affected stable branches, which you can find here [3].
I missed the fact that pip uses build isolation by default, which means it creates a custom venv with build dependencies installed before installing a package. Those build dependencies are not capped by default. I have proposed a follow-up [1] to address this. However, I don't know if this will be sufficient since tools like tox also install packages and the devstack `pip_install` function obviously does not affect those. I don't currently have a good idea how to resolve these issues, since the only constraints mechanism we have for build-time dependencies is the `build-system.requires` setting in `pyproject.toml`. While we could pin setuptools for openstack projects on old stable branches (tedious though it would be), we are likely to run into chicken-and-egg cross-dependency problems that prevent us actually merging these fixes and, worse, we have no way to fix non- OpenStack dependencies. I've started a discussion with the setuptools folks [1] but previous experience suggests we are unlikely to get much traction there (I'd love to be proven wrong though). If anyone else has good ideas I'd like to hear them. Stephen [1] https://review.opendev.org/c/openstack/devstack/+/976121
Cheers, Stephen
[1] https://github.com/k-orc/openstack-resource-controller/actions/runs/21818364... [2] https://review.opendev.org/q/project:openstack/pbr+topic:pep517 [3] https://review.opendev.org/q/topic:%22pin-setuptools%22+owner:stephenfin@red...
On 2026-02-09 15:08:06 +0000 (+0000), Stephen Finucane wrote: [...]
I missed the fact that pip uses build isolation by default, which means it creates a custom venv with build dependencies installed before installing a package. Those build dependencies are not capped by default.
I have proposed a follow-up [1] to address this. However, I don't know if this will be sufficient since tools like tox also install packages and the devstack `pip_install` function obviously does not affect those. I don't currently have a good idea how to resolve these issues, since the only constraints mechanism we have for build-time dependencies is the `build-system.requires` setting in `pyproject.toml`. While we could pin setuptools for openstack projects on old stable branches (tedious though it would be), we are likely to run into chicken-and-egg cross-dependency problems that prevent us actually merging these fixes and, worse, we have no way to fix non-OpenStack dependencies. [...]
Yes, in the past we considered it impossible to effectively pin SetupTools everywhere for essentially that reason (or related ones, like tools vendoring newer versions of it themselves). The emergence of pyproject.toml has changed that calculus more recently, but obviously only for projects/branches/versions where it's used already. -- Jeremy Stanley
On Tue, Feb 10, 2026 at 2:09 AM Stephen Finucane <stephenfin@redhat.com> wrote: [snip] I missed the fact that pip uses build isolation by default, which means
it creates a custom venv with build dependencies installed before installing a package. Those build dependencies are not capped by default.
Yes, that's a good call out. I don't think pinning Horizon will be sufficient because of that pip behaviour. The error I am seeing at the moment looks like this (sorry for the long paste, but I want to show the whole chain): Step 9/9 : RUN ln -s horizon-source/* horizon && sed -e "/^horizon===/d" -i requirements/upper-constraints.txt && python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /horizon "setuptools<82.0.0" && mkdir -p /etc/openstack-dashboard && cp -r /horizon/openstack_dashboard/conf/* /etc/openstack-dashboard/ && cp /horizon/openstack_dashboard/local/local_settings.py.example /etc/openstack-dashboard/local_settings.py && local_settings=$(python -c 'import os;import openstack_dashboard;settings=os.path.dirname(openstack_dashboard.__file__) + "/local/local_settings.py";print(settings);') && rm -f $local_settings && ln -s /etc/openstack-dashboard/local_settings.py $local_settings && mkdir -p /etc/openstack-dashboard/local_settings.d && local_settings_d=$(python -c 'import os;import openstack_dashboard;settings_d=os.path.dirname(openstack_dashboard.__file__) + "/local/local_settings.d";print(settings_d);') && rm -rf $local_settings_d && ln -s /etc/openstack-dashboard/local_settings.d $local_settings_d && cp /horizon/manage.py /var/lib/kolla/venv/bin/manage.py && if [ "$(ls /plugins)" ]; then python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /plugins/* "setuptools<82.0.0"; fi && for locale in /var/lib/kolla/venv/lib/python3/site-packages/*/locale; do (cd ${locale%/*} && /var/lib/kolla/venv/bin/django-admin compilemessages) done && chmod 644 /usr/local/bin/kolla_extend_start ---> Running in 895340bd989f Processing ./horizon Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Feb 09 11:01:03 master Requirement already satisfied: setuptools<82.0.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (81.0.0) Requirement already satisfied: pbr>=5.5.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (from horizon==25.6.0.dev84) (7.0.3) Feb 09 11:01:03 master Requirement already satisfied: Babel>=2.6.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (from horizon==25.6.0.dev84) (2.17.0) Collecting Django<4.3,>=4.2 (from horizon==25.6.0.dev84) Downloading django-4.2.25-py3-none-any.whl.metadata (4.2 kB) Collecting django-compressor>=4.4 (from horizon==25.6.0.dev84) Downloading django_compressor-4.5.1-py2.py3-none-any.whl.metadata (5.0 kB) Collecting django-debreach>=1.4.2 (from horizon==25.6.0.dev84) Downloading django_debreach-2.1.0-py3-none-any.whl.metadata (3.7 kB) │ exit code: 1 ╰─> [26 lines of output] Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> main() ~~~~^^ File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main json_out["return_val"] = hook(**hook_input["kwargs"]) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel return hook(config_settings) File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=[]) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires self.run_setup() ~~~~~~~~~~~~~~^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 520, in run_setup super().run_setup(setup_script=setup_script) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup exec(code, locals()) ~~~~^^^^^^^^^^^^^^^^ File "<string>", line 1, in <module> File "/tmp/pip-install-rgkzbtcj/xstatic-angular-schema-form_9a119b9343f1471a806215e0133ced88/xstatic/__init__.py", line 1, in <module> __import__('pkg_resources').declare_namespace(__name__) ~~~~~~~~~~^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'pkg_resources' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed to build 'XStatic-Angular-Schema-Form' when getting requirements to build wheel That paste has been lightly edited for clarity and contains a setuptools pin in the docker command RUN line for my attempts to resolve this. So I think somewhere in the dependency chain of 'XStatic-Angular-Schema-Form' is a problematic usage of setuptools.pkg_resources, I think in XStatic-Angular-Schema-Form itself? Interestingly, that's an Openstack project (one of the pypi maintainers is "openstack-ci") with exactly one release on pypi... From 2016. Michael
Michael, there is the fix already merged: https://review.opendev.org/c/openstack/kolla/+/976099 On Mon, Feb 9, 2026 at 9:45 PM Michael Still <mikal@stillhq.com> wrote:
On Tue, Feb 10, 2026 at 2:09 AM Stephen Finucane <stephenfin@redhat.com> wrote:
[snip]
I missed the fact that pip uses build isolation by default, which means
it creates a custom venv with build dependencies installed before installing a package. Those build dependencies are not capped by default.
Yes, that's a good call out. I don't think pinning Horizon will be sufficient because of that pip behaviour. The error I am seeing at the moment looks like this (sorry for the long paste, but I want to show the whole chain):
Step 9/9 : RUN ln -s horizon-source/* horizon && sed -e "/^horizon===/d" -i requirements/upper-constraints.txt && python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /horizon "setuptools<82.0.0" && mkdir -p /etc/openstack-dashboard && cp -r /horizon/openstack_dashboard/conf/* /etc/openstack-dashboard/ && cp /horizon/openstack_dashboard/local/local_settings.py.example /etc/openstack-dashboard/local_settings.py && local_settings=$(python -c 'import os;import openstack_dashboard;settings=os.path.dirname(openstack_dashboard.__file__) + "/local/local_settings.py";print(settings);') && rm -f $local_settings && ln -s /etc/openstack-dashboard/local_settings.py $local_settings && mkdir -p /etc/openstack-dashboard/local_settings.d && local_settings_d=$(python -c 'import os;import openstack_dashboard;settings_d=os.path.dirname(openstack_dashboard.__file__) + "/local/local_settings.d";print(settings_d);') && rm -rf $local_settings_d && ln -s /etc/openstack-dashboard/local_settings.d $local_settings_d && cp /horizon/manage.py /var/lib/kolla/venv/bin/manage.py && if [ "$(ls /plugins)" ]; then python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /plugins/* "setuptools<82.0.0"; fi && for locale in /var/lib/kolla/venv/lib/python3/site-packages/*/locale; do (cd ${locale%/*} && /var/lib/kolla/venv/bin/django-admin compilemessages) done && chmod 644 /usr/local/bin/kolla_extend_start
---> Running in 895340bd989f Processing ./horizon Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Feb 09 11:01:03 master Requirement already satisfied: setuptools<82.0.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (81.0.0) Requirement already satisfied: pbr>=5.5.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (from horizon==25.6.0.dev84) (7.0.3) Feb 09 11:01:03 master Requirement already satisfied: Babel>=2.6.0 in ./var/lib/kolla/venv/lib/python3.13/site-packages (from horizon==25.6.0.dev84) (2.17.0) Collecting Django<4.3,>=4.2 (from horizon==25.6.0.dev84) Downloading django-4.2.25-py3-none-any.whl.metadata (4.2 kB) Collecting django-compressor>=4.4 (from horizon==25.6.0.dev84) Downloading django_compressor-4.5.1-py2.py3-none-any.whl.metadata (5.0 kB) Collecting django-debreach>=1.4.2 (from horizon==25.6.0.dev84) Downloading django_debreach-2.1.0-py3-none-any.whl.metadata (3.7 kB) │ exit code: 1 ╰─> [26 lines of output] Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> main() ~~~~^^ File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main json_out["return_val"] = hook(**hook_input["kwargs"]) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/kolla/venv/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel return hook(config_settings) File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=[]) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires self.run_setup() ~~~~~~~~~~~~~~^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 520, in run_setup super().run_setup(setup_script=setup_script) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pip-build-env-n_96tptu/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup exec(code, locals()) ~~~~^^^^^^^^^^^^^^^^ File "<string>", line 1, in <module> File "/tmp/pip-install-rgkzbtcj/xstatic-angular-schema-form_9a119b9343f1471a806215e0133ced88/xstatic/__init__.py", line 1, in <module> __import__('pkg_resources').declare_namespace(__name__) ~~~~~~~~~~^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'pkg_resources' [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed to build 'XStatic-Angular-Schema-Form' when getting requirements to build wheel
That paste has been lightly edited for clarity and contains a setuptools pin in the docker command RUN line for my attempts to resolve this.
So I think somewhere in the dependency chain of 'XStatic-Angular-Schema-Form' is a problematic usage of setuptools.pkg_resources, I think in XStatic-Angular-Schema-Form itself? Interestingly, that's an Openstack project (one of the pypi maintainers is "openstack-ci") with exactly one release on pypi... From 2016.
Michael
-- Regards, Maksim Malchuk
participants (9)
-
Bartosz Bezak
-
Jeremy Stanley
-
Maksim Malchuk
-
Michael Still
-
Radomir Dopieralski
-
Sean Mooney
-
Stephen Finucane
-
Takashi Kajinami
-
Yatin Karel