I am running 3 node controllers in multi-node deployment using the 2023.1 release and life was good until one of my controller nodes died. after losing one of the controller node I have started seeing following error in keystone logs 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application [None req-95a83b6e-dacc-45d4-8d1f-4d5c979f79e6 d58d8e432da74d8ba4416054d1e36a84 d1ef63c18c6648faa56643a942216311 - - default default] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 89, in unpack 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.crypto.decrypt(token.encode('utf-8')) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/cryptography/fernet.py", line 210, in decrypt 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise InvalidToken 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application cryptography.fernet.InvalidToken 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 99, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.token_formatter.validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 173, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application serialized_payload = self.unpack(token) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 91, in unpack 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise exception.ValidationError( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last): 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application rv = self.dispatch_request() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 145, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application token = self._validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.get_or_create( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application with Lock( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self._enter() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application return self.creator() 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application created_value = creator( 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 158, in _validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application issued_at, expires_at) = self.driver.validate_token(token_id) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 101, in validate_token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application raise exception.TokenNotFound(e) 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token 2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application I have rotated the fernet key and also verified in docker container and md5sum is match and date/time also very accurate. I am not sure what else I should look for. (keystone)[root@os1-ctrl01 fernet-keys]# md5sum /etc/keystone/fernet-keys/* 4874cf812b73f18cde82f1d00ea681e1 /etc/keystone/fernet-keys/0 e1a40f4cd58cfbc9b3b5af7305a16fc8 /etc/keystone/fernet-keys/214 0f45eeb318e3c4bfb97bb07da790329a /etc/keystone/fernet-keys/215 (keystone)[root@os1-ctrl03 /]# md5sum /etc/keystone/fernet-keys/* 4874cf812b73f18cde82f1d00ea681e1 /etc/keystone/fernet-keys/0 e1a40f4cd58cfbc9b3b5af7305a16fc8 /etc/keystone/fernet-keys/214 0f45eeb318e3c4bfb97bb07da790329a /etc/keystone/fernet-keys/215