Keystone "TokenNotFound: Failed to validate token"
Hi all, I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1]. From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs. Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451
Hi all, has really no one else seen this behavior? Keystone logs are still flooded with TokenNotFound: Failed to validate token (≈18k/day). I already verified: * Clock skew and memcached → fine * Disabled token cache (revocation_cache_time = 0) → no effect * keystone_authtoken config → consistent * Nova/Cinder show also InvalidToken at the same time Keystone logs TokenNotFound (but the request ID don't match) Any ideas on what could trigger this in 2024.1 / Keystone 25.0.1 would be very welcome. Best regards, Marc Am Donnerstag, August 14, 2025 09:53 CEST, schrieb "Marc Vorwerk" <marc+openstack@marc-vorwerk.de>: Hi all, I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1]. From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs. Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451
I don’t have an answer, I just wanted to share that I see one single message like yours for today. I haven’t checked older logs yet, I’ll take another look when I find the time. This cluster (Caracal) is not that heavily used, though. But I have access to another cluster which is larger than ours and with way more traffic. I will take a look there as well. But they’re running on Victoria… Zitat von Marc Vorwerk <marc+openstack@marc-vorwerk.de>:
Hi all, has really no one else seen this behavior? Keystone logs are still flooded with TokenNotFound: Failed to validate token (≈18k/day). I already verified: * Clock skew and memcached → fine * Disabled token cache (revocation_cache_time = 0) → no effect * keystone_authtoken config → consistent * Nova/Cinder show also InvalidToken at the same time Keystone logs TokenNotFound (but the request ID don't match) Any ideas on what could trigger this in 2024.1 / Keystone 25.0.1 would be very welcome. Best regards, Marc
Am Donnerstag, August 14, 2025 09:53 CEST, schrieb "Marc Vorwerk" <marc+openstack@marc-vorwerk.de>: Hi all,
I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1].
From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs.
Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451
I didn't find any of these messages in the larger cloud, at all. The only thing I found in our own is this line matching the timestamp of the validation failure: 192.168.160.9 - - [19/Aug/2025:21:24:39 +0200] "GET /v3/9f743e485e3f477c8935dd29903a1147/volumes/detail HTTP/1.1" 401 114 "-" "cinder-csi-plugin/v1.32.0 gophercloud/v2.5.0" 130959(us) But I don't have an explanation yet why it would fail. I don't manage our k8s infrastructure so I have no idea if anything failed on that side. But I'll try to find out if this had any impact. But again, I found only 3 occurences in total over the last few of weeks, so it doesn't seem to be a real issue for us. Zitat von Eugen Block <eblock@nde.ag>:
I don’t have an answer, I just wanted to share that I see one single message like yours for today. I haven’t checked older logs yet, I’ll take another look when I find the time. This cluster (Caracal) is not that heavily used, though. But I have access to another cluster which is larger than ours and with way more traffic. I will take a look there as well. But they’re running on Victoria…
Zitat von Marc Vorwerk <marc+openstack@marc-vorwerk.de>:
Hi all, has really no one else seen this behavior? Keystone logs are still flooded with TokenNotFound: Failed to validate token (≈18k/day). I already verified: * Clock skew and memcached → fine * Disabled token cache (revocation_cache_time = 0) → no effect * keystone_authtoken config → consistent * Nova/Cinder show also InvalidToken at the same time Keystone logs TokenNotFound (but the request ID don't match) Any ideas on what could trigger this in 2024.1 / Keystone 25.0.1 would be very welcome. Best regards, Marc
Am Donnerstag, August 14, 2025 09:53 CEST, schrieb "Marc Vorwerk" <marc+openstack@marc-vorwerk.de>: Hi all,
I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1].
From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs.
Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451
participantes (2)
-
Eugen Block
-
Marc Vorwerk