Hi Jake, I had more urgent issues to fix and have not looked more to this one, except that today I checked the problem is already present in Yoga on our production cluster so it is not linked to the Antelope upgrade, conversely to what I thought. This error happens when deleting Magnum clusters and the clusters were also created by Magnum. I have no clue what is done in which order, I don't do anything special as far as I can tell. The full Antelope traceback corresponding to the problem is below (the Yoga one looks very similar with different paths due to different Python versions). Best regards, Michel -------------------------------------------- 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Failed to delete trust: keystoneauth1.exceptions.htt p.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9cce561e-1b7e-4b30-84ae-bddbb2a3b01a) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone Traceback (most recent call last): 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/magnum/common/keystone.py", line 246, in delete_trust 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone client.trusts.delete(cluster.trust_id) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneclient/v3/contrib/trusts.py", line 102, in delete 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return super(TrustManager, self).delete(trust_id=base.getid(trust)) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneclient/base.py", line 86, in func 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return f(*args, **new_kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneclient/base.py", line 479, in delete 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return self._delete( 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneclient/base.py", line 246, in _delete 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone resp, body = self.client.delete(url, **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 410, in delete 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return self.request(url, 'DELETE', **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 554, in request 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return self.session.request(url, method, **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 790, in request 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone auth_headers = self.get_auth_headers(auth) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1201, in get_auth_headers 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return auth.get_headers(self, **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/plugin.py", line 106, in get_headers 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone token = self.get_token(session) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return self.get_access(session).auth_token 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone self.auth_ref = self.get_auth_ref(session) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/v3/base.py", line 188, in get_auth_ref 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone resp = session.post(token_url, json=body, headers=headers, 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1149, in post 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone return self.request(url, 'POST', **kwargs) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone raise exceptions.from_response(resp, method, url) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9cce561e-1b7e-4b30-84ae-bddbb2a3b01a) 2024-05-31 08:45:51.920 280026 ERROR magnum.common.keystone 2024-05-31 08:45:51.955 280026 INFO magnum.common.cert_manager.barbican_cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/8b27c465-19b1-48c0-8018-001... from Barbican. 2024-05-31 08:45:51.956 280026 INFO barbicanclient.base [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Calculated Containers uuid ref: containers/8b27c465-19b1-48c0-8018-0018bb4be077 2024-05-31 08:45:51.965 280026 ERROR barbicanclient.client [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/8b27c465-19b1-48c0-8018-001...: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager Traceback (most recent call last): 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/magnum/common/cert_manager/barbican_cert_manager.py", line 198, in delete_cert 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager certificate_container = connection.containers.get(cert_ref) 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/v1/containers.py", line 543, in get 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager response = self._api.get(uuid_ref) 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 70, in get 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager return super(_HTTPClient, self).get(*args, **kwargs).json() 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager return self.request(url, 'GET', **kwargs) 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 63, in request 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager self._check_status_code(resp) 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 105, in _check_status_code 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager raise exceptions.HTTPClientError( 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. 2024-05-31 08:45:51.966 280026 ERROR magnum.common.cert_manager.barbican_cert_manager 2024-05-31 08:45:51.967 280026 WARNING magnum.conductor.handlers.common.cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Deleting certs is failed for Cluster 18e51e7c-0a31-4b4c-a4fd-8142ac9f877d: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. 2024-05-31 08:45:51.967 280026 INFO magnum.common.cert_manager.barbican_cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/e90f62f5-6df9-4d51-b3dd-61e... from Barbican. 2024-05-31 08:45:51.967 280026 INFO barbicanclient.base [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Calculated Containers uuid ref: containers/e90f62f5-6df9-4d51-b3dd-61e6289812de 2024-05-31 08:45:51.976 280026 ERROR barbicanclient.client [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/e90f62f5-6df9-4d51-b3dd-61e...: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager Traceback (most recent call last): 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/magnum/common/cert_manager/barbican_cert_manager.py", line 198, in delete_cert 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager certificate_container = connection.containers.get(cert_ref) 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/v1/containers.py", line 543, in get 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager response = self._api.get(uuid_ref) 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 70, in get 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager return super(_HTTPClient, self).get(*args, **kwargs).json() 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager return self.request(url, 'GET', **kwargs) 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 63, in request 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager self._check_status_code(resp) 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 105, in _check_status_code 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager raise exceptions.HTTPClientError( 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. 2024-05-31 08:45:51.977 280026 ERROR magnum.common.cert_manager.barbican_cert_manager 2024-05-31 08:45:51.978 280026 WARNING magnum.conductor.handlers.common.cert_manager [None req-d9125739-f2e0-404f-953e-fe461578f2fd - - - - - -] Deleting certs is failed for Cluster 18e51e7c-0a31-4b4c-a4fd-8142ac9f877d: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. Le 05/06/2024 à 15:44, Jake Yip a écrit :
On 29/5/2024 10:36 pm, Michel Jouvin wrote:
In Keystone log, I find with the same timestamp the following error that seems related to the one above:
----
2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ----
Wild guess, was the trust deleted before the secret? If so, how are you creating and deleting these secrets?
Are you able to trace where the calls are happening? Maybe some dependency is missing in the deletion pathway.
Regards, Jake