On Mon, 2026-02-09 at 15:08 +0000, Stephen Finucane wrote:
On Mon, 2026-02-09 at 11:50 +0000, Stephen Finucane wrote:
On Mon, 2026-02-09 at 18:49 +1100, Michael Still wrote:
Hey all,
I've just noticed that setuptools v82.0.0 released todayish and breaks a bunch of Kolla builds (and probably other things) by removing pkg_resources. https://setuptools.pypa.io/en/stable/history.html#v82-0-0 has further details of the release.
setuptools doesn't seem to be listed in upper-constraints, perhaps because it's often provided as a distro package?
Michael
We're seeing this on projects outside of opendev also, such as ORC [1]. The vast majority (though not all, as Takashi notes) of these are due to use of old versions of pbr. These issues are all resolved in recent pbr releases [2]. While we could bump the pbr constraint in upper- constraints for these stable branches, I think it would be wiser to pin setuptools as part of the devstack global venv creation since there were more users of pkg_resources that have only been removed in recent releases. I've proposed devstack changes to do just this for the affected stable branches, which you can find here [3].
I missed the fact that pip uses build isolation by default, which means it creates a custom venv with build dependencies installed before installing a package. Those build dependencies are not capped by default.
I have proposed a follow-up [1] to address this. However, I don't know if this will be sufficient since tools like tox also install packages and the devstack `pip_install` function obviously does not affect those. I don't currently have a good idea how to resolve these issues, since the only constraints mechanism we have for build-time dependencies is the `build-system.requires` setting in `pyproject.toml`. While we could pin setuptools for openstack projects on old stable branches (tedious though it would be), we are likely to run into chicken-and-egg cross-dependency problems that prevent us actually merging these fixes and, worse, we have no way to fix non- OpenStack dependencies.
I've started a discussion with the setuptools folks [1] but previous experience suggests we are unlikely to get much traction there (I'd love to be proven wrong though). If anyone else has good ideas I'd like to hear them.
Just a follow-up on this. I have had some engagement, though not yet from the person I suspect will ultimately make the call (jaraco). There are parallel discussions going on in [1] and [2]. [1] is my issue and asks to restore and simply freeze 'pkg_resources'. [2] is a re-opened issue that asks to publish 'pkg_resources' to PyPI. I'm heavily in favour of the former owing to complexities when 'pkg_resources' is a build-time dependency. Hopefully we'll get some traction here because the alternative, which I suspect will involve a combination of the existing devstack changes *plus* modifications to virtually all our 'tox.ini' / 'nox.py' files on stable branches *plus* changes to distributions like kolla, is going to an outrageous amount of work. Stephen [1] https://github.com/pypa/setuptools/issues/5174 [2] https://github.com/pypa/setuptools/issues/863
Stephen
[1] https://review.opendev.org/c/openstack/devstack/+/976121
Cheers, Stephen
[1] https://github.com/k-orc/openstack-resource-controller/actions/runs/21818364... [2] https://review.opendev.org/q/project:openstack/pbr+topic:pep517 [3] https://review.opendev.org/q/topic:%22pin-setuptools%22+owner:stephenfin@red...