I am running 3 node controllers in multi-node deployment using the 2023.1 release and life was good until one of my controller nodes died. 

after losing one of the controller node I have started seeing following error in keystone logs 

2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application [None req-95a83b6e-dacc-45d4-8d1f-4d5c979f79e6 d58d8e432da74d8ba4416054d1e36a84 d1ef63c18c6648faa56643a942216311 - - default default] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last):
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 89, in unpack
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self.crypto.decrypt(token.encode('utf-8'))
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/cryptography/fernet.py", line 210, in decrypt
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     raise InvalidToken
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application cryptography.fernet.InvalidToken
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last):
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 99, in validate_token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self.token_formatter.validate_token(token_id)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 173, in validate_token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     serialized_payload = self.unpack(token)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/token_formatters.py", line 91, in unpack
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     raise exception.ValidationError(
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application Traceback (most recent call last):
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     rv = self.dispatch_request()
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     resp = resource(*args, **kwargs)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     resp = meth(*args, **kwargs)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     ENFORCER.enforce_call(action='identity:validate_token')
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     subj_token_target_data = cls._extract_subject_token_target_data()
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     token = PROVIDER_APIS.token_provider_api.validate_token(
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     __ret_val = __f(*args, **kwargs)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 145, in validate_token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     token = self._validate_token(token_id)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return caller(func, *(extras + args), **kw)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self.get_or_create(
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     with Lock(
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self._enter()
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     generated = self._enter_create(value, createdtime)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     return self.creator()
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     created_value = creator(
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 158, in _validate_token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     issued_at, expires_at) = self.driver.validate_token(token_id)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/providers/fernet/core.py", line 101, in validate_token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application     raise exception.TokenNotFound(e)
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token
2025-11-04 01:13:14.514 25 ERROR keystone.server.flask.application

I have rotated the fernet key and also verified in docker container and md5sum is match and date/time also very accurate. I am not sure what else I should look for. 

(keystone)[root@os1-ctrl01 fernet-keys]# md5sum /etc/keystone/fernet-keys/*
4874cf812b73f18cde82f1d00ea681e1  /etc/keystone/fernet-keys/0
e1a40f4cd58cfbc9b3b5af7305a16fc8  /etc/keystone/fernet-keys/214
0f45eeb318e3c4bfb97bb07da790329a  /etc/keystone/fernet-keys/215

(keystone)[root@os1-ctrl03 /]# md5sum /etc/keystone/fernet-keys/*
4874cf812b73f18cde82f1d00ea681e1  /etc/keystone/fernet-keys/0
e1a40f4cd58cfbc9b3b5af7305a16fc8  /etc/keystone/fernet-keys/214
0f45eeb318e3c4bfb97bb07da790329a  /etc/keystone/fernet-keys/215