August 2025 - openstack-discuss - lists.openstack.org

Keystone "TokenNotFound: Failed to validate token"
by Marc Vorwerk 23 Aug '25

23 Aug '25

Hi all, I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1]. From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs. Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451

2 3

[release] Release countdown for week R-5
by Thierry Carrez 22 Aug '25

22 Aug '25

Development Focus ----------------- We are getting close to the end of the 2025.2 "Flamingo" cycle! Next week on August 28th, 2025 is the Flamingo-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the 2026.1 "Gazpacho" cycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on August 28th, 2025. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/2025.2 branching request with the review. As an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… August 28th, 2025 is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be raised as a Feature Freeze Exception (FFE) and approved by the team PTL. Finally, feature freeze is also a good time for submitting a first version of your cycle-highlights. Cycle highlights are the raw data that helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Flamingo-3 milestone (feature freeze): August 28th, 2025 (R-5 week) RC1 deadline: September 12th, 2025 (R-3 week) Final RC deadline: September 25th, 2025 (R-1 week) 2025.2 "Flamingo" final release: October 1st, 2025 OpenInfra Summit Europe: October 17th-19th, 2025 -- Thierry Carrez (ttx)

1 0

[blazar] Blazar-dashboard contains non-free files (ie: minified Javascripts)
by Thomas Goirand 22 Aug '25

22 Aug '25

Hi, I've been trying to package Blazar, all is already in Debian, but I can't get blazar-dashboard in, because upstream source contains non-free files. Namely: https://github.com/openstack/blazar-dashboard/blob/master/blazar_dashboard/… is a minifed version of a much more complex package: https://github.com/apexcharts/apexcharts.js/tree/v3.31.0 Reminder: minified Javascripts are considered by most distributions (this includes Red Hat, Ubuntu, and in my case Debian) as non-free files that cannot be shipped in the distribution without the underlying source code, and packages must build from the not minified version. Could the Blazar team work on un-vendoring apexcharts.min.js, possibly removing it completely, or replacing it with a source-full version? Cheers, Thomas Goirand (zigo)

4 6

[neutron] Drivers meeting canceled Friday, August 22nd
by Brian Haley 21 Aug '25

21 Aug '25

Hello Neutrinos: I am going to cancel tomorrow's Drivers meetings as there will not be quorum. Just a reminder there are still open specs that need updates and/or review, list is at [0]. Have a nice weekend! -Brian [0] https://review.opendev.org/q/status:open+project:openstack/neutron-specs

1 0

[all] Keeping old unmaintained branches open for longer time, but with no CI, instead of EOL tags
by Thomas Goirand 21 Aug '25

21 Aug '25

Hi, It's our policy that we maintain only a few branches, currently, from Caracal to Epoxy (so, 3 stable branches). Currently, when a Stable branch becomes unsuported, we rename that branch as: unmaintained/2023.1 which is fine, as we can still commit there. Though after some times, the branches are deleted, and instead, we just do a tag such as: zed-EOL Jeremy Stanley just wrote to me on IRC: "workflows, processes and policies are built around many years of an assumption that unused branches will be deleted" Keeping unmaintained branches "has put a lot of additional strain on project maintainers and our systems" (still quoting Jeremy). Though each time there's a new (grave?) security problem, this topic comes back, as we still have no solution for these cases. As a downstream, I still need to write backports for these important patches, sometimes long after the branches are EOL. Red Hat too... While I agree with all what Jeremy told me on IRC, and I don't think we should attempt to maintain the CI for too long (as this takes too much of our time), I still believe we should keep a way to share patches. Downstream (like myself for Debian) run their own CI based on packages, and it's always better if we can share patches. It doesn't have to be with a maintained CI. Just having opened branches would be enough. Your thoughts? Cheers, Thomas Goirand (zigo)

6 6

[all][elections][ptl][tc] Combined PTL/TC 2026.1 cycle Election Nominations End
by Ian Y. Choi 20 Aug '25

20 Aug '25

The PTL and TC Nomination period is now over. The official candidate lists for PTLs [0] and TC seats [1] are available on the election website. -- PTL Election Details -- There are 4 projects without candidates, so according to this resolution[2], the TC will have to decide how the following projects will proceed: Monasca, OpenStack_Charms, Venus, Vitrage There are 1 project that will have elections: Horizon. -- TC Election Details -- The official candidate list is available on the election website[1]. Now begins the campaigning period where candidates and electorate may debate their statements. Polling will start Aug 27, 2025 23:45 UTC. Thank you, [0] https://governance.openstack.org/election/#ptl-candidates [1] https://governance.openstack.org/election/#tc-candidates [2] https://governance.openstack.org/resolutions/20141128-elections-process-for…

1 0

OSSN-0094: Ensuring Volume Safety with Nova and Watcher
by Jay Faulkner 20 Aug '25

20 Aug '25

OSSN-0094: Ensuring Volume Safety with Nova and Watcher == Summary == A vulnerability has been identified in OpenStack Nova and OpenStack Watcher in conjunction with volume swap operations performed by the Watcher service. Under specific circumstances, this can lead to a situation where two Nova libvirt instances could reference the same block device, allowing accidental information disclosure to the unauthorized instance. == Affected Services / Software == Services: Nova,Watcher Releases: all supported releases == Discussion == The issue occurs when Watcher's zone migration strategy performs the following sequence of events: 1. Watcher initiates a volume swap using Nova's internal-only volume swap API 2. Watcher initiates a live migration of the same instance 3. In some error cases connection details may have failed to update storage references. These invalid details are used during the live migration. === Required Access === The swap volume, live migration and all Watcher APIs are admin only so with default policy is only possible to create the inconsistent state described in this OSSN if you have admin rights on the relevant OpenStack project. === Further Watcher Hardening === The Watcher service, when first created, often implemented its own means to perform operations. Many of those operations can now be done natively via other OpenStack services. In the specific context of OSSN-0094, the ability to migrate Cinder volumes between storage backends is such an example. Additionally, the Cinder volume migration in Watcher created a new Keystone user with the admin role assigned for the instance owners' project and then used that user to perform API requests on behalf of the project. This code has been removed. Finally, due to limited error handling and no validation that the objects involved were migrated properly, some error scenarios could have led to a source volume being deleted despite not having been migrated properly. === Resolution === Nova will now reject any request to swap a volume that has an empty migration status, effectively restricting the usage of this API to Cinder. This brings the API validation in line with the documentation. Watchers internal implementation of swap volume has been deleted and updated to use Cinder's native volume migration as a replacement. Watcher no longer creates temporary Keystone users in normal operation. === Patches === Patches for Nova and Watcher have been backported to all supported stable branches and committed to master branch. stable/2025.1: * Watcher: https://review.opendev.org/c/openstack/watcher/+/957770 * Nova: https://review.opendev.org/c/openstack/nova/+/957759 stable/2024.2: * Watcher: https://review.opendev.org/c/openstack/watcher/+/957773 * Nova: https://review.opendev.org/c/openstack/nova/+/957762 stable/2024.1: * Watcher: https://review.opendev.org/c/openstack/watcher/+/957774 * Nova: https://review.opendev.org/c/openstack/nova/+/957764 == Recommended Actions == * Operators using Watcher's zone migration strategy should apply the provided Watcher and Nova patches as soon as possible. * Operators should refrain from using the swap volume migration action in Watcher. The compatibility code for swap volume that uses a Cinder-based migration may be removed in a future API version. * Operators should audit all users with the admin role and ensure no temporary Watcher-created users remain. * Operators using custom policy for volume attachment (''/servers/{server_id}/os-volume_attachments/{volume_id}'') or live migration API should review the state of existing instances which have had volume migrations. Any instance in an inconsistent state can be resolved by hard rebooting the instance using Nova's API. == Contacts / References == * Author: Sean Mooney <smooney(a)redhat.com>, Jay Faulkner <jay(a)jvf.cc> * This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0094 * Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/2112187 * Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org * OpenStack Security Project : https://launchpad.net/~openstack-ossg * CVE: None

1 0

[vitrage][tc] Project status
by Takashi Kajinami 20 Aug '25

20 Aug '25

Hi, Some of you might be aware of this, but tempest jobs in vitrage repo have been broken for some time (I remember I saw consistent failures early in this cycle and I'm afraid that the situation hasn't been improved.) I spent some time to dig into these failures and found out a few problems. However I'm afraid I can't fix these all with my limited knowledge about vitrage. # I've helped its project maintenance for some time but I've never looked into # its implementations quite in detail. So I'm sharing my observations here hoping someone who know better about it can look into these. 1. Usage of removed oslo_context.context.RequestContext interfaces This is the most trivial and can be fixed easily. https://review.opendev.org/c/openstack/vitrage/+/957004/2/vitrage/api/hooks… 2. Usage of asyncore modules from pysnmp-lextudio Recent release of pysnmp-lextudio no longer provides these modules, likely due to removal of asyncore in Python 3.12 . I was drafting the migration to asyncio version in https://review.opendev.org/c/openstack/vitrage/+/957004 but learned simple replacement does not work and we need more careful assessment. This might be the most tricky problem atm. 3. Some tempest test failures I can no longer see old CI results, but as far as I'm aware, CI jobs were blocked by a few tests consistently failing early in this cycle so we may have to dig into these, too. You may find some tempest tests failures in https://review.opendev.org/c/openstack/vitrage/+/957004 though I'm unsure if these are related to broken snmp service, which is caused by incomplete migration to asyncio (which I described in 2). I'm adding the tc tag, to notify them of the current status. Thank you, Takashi Kajinami -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 2

[swift] Unmaintained branch cleanup
by Tim Burke 20 Aug '25

20 Aug '25

Swift currently has six unmaintained branches: - unmaintained/2023.1 - unmaintained/zed - unmaintained/yoga - unmaintained/xena - unmaintained/wallaby - unmaintained/victoria Half of them have not been updated this year. For all of them, the only patches since the transition to unmaintained status were to try to maintain CI. While two of them have open patches, they all seem to revolve around maintaining CI and none of them currently pass. I propose transitioning all of these to end-of-life. If you have any objections, please respond here on the mailing list or at https://review.opendev.org/c/openstack/releases/+/958101 which makes the change. Tim

1 0

[horizon] Today's weekly meeting is cancelled
by Tatiana Ovchinnikova 20 Aug '25

20 Aug '25

Hello team, A reminder that today's meeting is cancelled since more than a half of the team is busy with downstream activities. With kind regards, Tatiana

1 0