Dear everyone, I run a small OpenStack cloud on Debian Buster, using standard distro packages - i.e. Rocky. My Keystone is configured as a federated-identity Service Provider, with the IdP accessed using OpenID. Having got things working successfully using local users I went on to testing federation - and found out that using the OpenStack CLI to get a scoped token from an unscoped one reports a server error 500, and logging in to Horizon shows usage and event data couldn't be retrieved. On the server side, in both cases Keystone log shows the following: INFO keystone.common.wsgi [req-foo bar baz - Federated default] POST https://osc.example.com:5000/v3/auth/tokens ERROR keystone.common.wsgi [req-foo bar baz - Federated default] identity_provider failed validation: <function FederatedCredential.<lambda> at 0xdeadbeef>: ValueError: identity_provider failed validation: <function FederatedCredential.<lambda> at 0xdeadbeef> ERROR keystone.common.wsgi Traceback (most recent call last): ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__ ERROR keystone.common.wsgi result = method(req, **params) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/auth/controllers.py", line 67, in authenticate_for_token ERROR keystone.common.wsgi self.authenticate(request, auth_info, auth_context) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/auth/controllers.py", line 236, in authenticate ERROR keystone.common.wsgi auth_info.get_method_data(method_name)) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/auth/plugins/token.py", line 46, in authenticate ERROR keystone.common.wsgi PROVIDERS.identity_api ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/auth/plugins/mapped.py", line 101, in handle_scoped_token ERROR keystone.common.wsgi send_notification(taxonomy.OUTCOME_SUCCESS) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/keystone/notifications.py", line 685, in send_saml_audit_notification ERROR keystone.common.wsgi user=user_id, groups=group_ids) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/pycadf/credential.py", line 84, in __init__ ERROR keystone.common.wsgi setattr(self, FED_CRED_KEYNAME_IDENTITY_PROVIDER, identity_provider) ERROR keystone.common.wsgi File "/usr/lib/python3/dist-packages/pycadf/cadftype.py", line 66, in __set__ ERROR keystone.common.wsgi (self.name, self.func)) ERROR keystone.common.wsgi ValueError: identity_provider failed validation: <function FederatedCredential.<lambda> at 0xdeadbeef> i.e. the request has succeeded but then things fall over when an audit notification is to be sent. Having poked around the sources of keystone.notifications.send_saml_audit_notifications() and the code it references, I found out the following: - the lambda function which triggers the error checks if 'identity_provider' is a six string type; - when this error occurs the value of 'identity_provider' is indeed the name of my IdP - but as *bytes* rather than str! - this doesn't happen every time this IdP name is used - if I add a simple identity_provider = identity_provider.decode('utf-8') to the relevant function I start getting errors suggesting that under some circumstances, 'identity_provider' is str as it should be. All in all, it seems like this particular bit of Keystone code in Rocky does not properly support Python3. Unfortunately an upgrade to a newer OpenStack version is non-trivial for me for several reasons, only some of which are under my control - and the stop-gap measure of having patched the relevant function with "if identity_provider is bytes, decode it to str" means I've had to put the relevant Debian package on hold, thus blocking potential updates. Therefore, I look forward to hearing from you any and all ideas which might help address this problem in less radical fashion. Thank you in advance! -- MS