o/

pip 25.3 was released on Friday (who does Friday releases?!) and had
removed support for the legacy `setup.py develop` editable install
method in setuptools [1]. Now, thankfully most services seem to have
listened to me [2][3] and the shouty warnings from setuptools/pip and
migrated away from 'wsgi_scripts'. However, there is a good chance that
older stable branches are missing these migration patches and
consequently now broken. For example, it seems keystone is missing
these changes on stable/2024.2, and attempts to deploy that branch
currently fail with:

   Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500)

which, upon further inspection, will turn out to be a missing wsgi
script:

   devstack@keystone.service[40466]: failed to open python file /opt/stack/data/venv/bin/keystone-wsgi-public

We have already seen this pop up on the cluster-api-provider-openstack
(CAPO) project [4], which is using the old stable/2024.2 (Dalmatian)
release.

I have proposed fixes to stable/2025.1 [5] and stable/2024.2 [6] to
constrain pip on those older branches. This should fix the issue for
everyone, provided you're not using a devstack plugin that re-updates
pip. Individual services may also wish to backport the patches that
added '$service.wsgi' modules if possible, to fix this for non-devstack
use cases, but this won't be possible in all cases.

Hope this helps,
Stephen

[1] https://pip.pypa.io/en/stable/news/#v25-3
[2] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/DOJZL5WZJMN4NEAD5GLIQZ455D2NUREY/
[3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/5RQGZZMCKKHAVH4OLSJCLBQNIHYDPGZD/
[4] https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/2801
[5] https://review.opendev.org/c/openstack/devstack/+/965092
[6] https://review.opendev.org/c/openstack/devstack/+/965093