Magnum error after Antelope upgrade
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
----
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic ---
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit :
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi,
I am still stucked with this problem... Any other suggestion?
Cheers,
Michel
Le 29/04/2024 à 13:08, Michel Jouvin a écrit :
Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit :
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi.
With Antelope and 2 domains, same error. With Antelope and only one domain, no error. Maybe the same for you ?
Franck VEDEL
Le 30 avr. 2024 à 10:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi,
I am still stucked with this problem... Any other suggestion?
Cheers,
Michel
Le 29/04/2024 à 13:08, Michel Jouvin a écrit :
Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit :
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi,
I also have two domains. Default and one LDAP Active Directory. And had no issue. But if I recall correctly, I upgraded from yoga to zed and not directly to antelope.
Could it be related to the deprecation of the _member_ role?
Cheers, Oliver
Von meinem iPhone gesendet
Am 30.04.2024 um 11:37 schrieb Franck VEDEL franck.vedel@univ-grenoble-alpes.fr:
Hi.
With Antelope and 2 domains, same error. With Antelope and only one domain, no error. Maybe the same for you ?
Franck VEDEL
Le 30 avr. 2024 à 10:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi,
I am still stucked with this problem... Any other suggestion?
Cheers,
Michel
Le 29/04/2024 à 13:08, Michel Jouvin a écrit : Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit : Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
Hi,
I have ~5 (user) domains. I don't think that the direct upgrade from Yoga to Antelope is the issue, as it is supposed to be supported (even if somewhat experimental as it was the first SLURP release, see some previous discussion on the list about it). I'll check about the deprecation of _member_ role, may be I missed something important...
Cheers,
Michel
Le 30/04/2024 à 13:24, Oliver Weinmann a écrit :
Hi,
I also have two domains. Default and one LDAP Active Directory. And had no issue. But if I recall correctly, I upgraded from yoga to zed and not directly to antelope.
Could it be related to the deprecation of the _member_ role?
Cheers, Oliver
Von meinem iPhone gesendet
Am 30.04.2024 um 11:37 schrieb Franck VEDEL franck.vedel@univ-grenoble-alpes.fr:
Hi.
With Antelope and 2 domains, same error. With Antelope and only one domain, no error. Maybe the same for you ?
Franck VEDEL
Le 30 avr. 2024 à 10:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi,
I am still stucked with this problem... Any other suggestion?
Cheers,
Michel
Le 29/04/2024 à 13:08, Michel Jouvin a écrit :
Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit :
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
Am 29.04.2024 um 12:33 schrieb Michel Jouvin michel.jouvin@ijclab.in2p3.fr:
Hi,
In a cloud where all services have been upgraded from Yoga to Antelope, I tried to upgrade Magnum but after the (successful) upgrade, Magnum reports "keystone client errors" when trying to retrieve the status of existing clusters or to create new ones (deletion of existing ones seem to work).
The typical error is:
2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic [-] Skip pulling data from cluster 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected keystone clie nt error occurred: Forbidden (HTTP 403): magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic Traceback (most recent call last): 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/exception.py", line 57, in wrapped 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return func(*args, **kw) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/magnum/common/clients.py", line 166, in barbican 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic self._barbican = barbicanclient.Client(session=session, 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/client.py", line 186, in Client 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return client_class(session=session, *args, **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 47, in __init__ 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic normalized_microversion = self._get_max_supported_version( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 88, in _get_max_supported_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic min_ver, max_ver = self._get_min_max_server_supported_microversion( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 119, in _get_min_max_server_supporte d_microversion 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self._get_min_max_version(session, endpoint, '1.1') 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", line 125, in _get_min_max_version 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = discover.get_version_data( 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", line 107, in get_version_data 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp = session.get(url, headers=headers, authenticated=authenticated) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic return self.request(url, 'GET', **kwargs) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 986, in request 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic raise exceptions.from_response(resp, method, url) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic
Did I miss something in the upgrade procedure?
Thanks in advance for any help. Best regards,
Michel
I had a new look at the problem and found nothing to help as far as I can say... The keystone client error mentions a 403 status but I cannot find any trace of a matching request on the keystone server. All the requests from the Magnum/Heat server are completing with a 200 status according to the keystone (httpd) server logs...
The only error found in keystone logs is the "infamous" "Truncating password to algorithm specific maximum length 72 characters." which started to appear in Xena or Yoga, I don't remember, is happening all the time without any croncrete impact on services up to know... And it should not explain a 403 error that suggests more something related to the httpd configuration...
Cheers,
Michel
Le 30/04/2024 à 13:35, Michel Jouvin a écrit :
Hi,
I have ~5 (user) domains. I don't think that the direct upgrade from Yoga to Antelope is the issue, as it is supposed to be supported (even if somewhat experimental as it was the first SLURP release, see some previous discussion on the list about it). I'll check about the deprecation of _member_ role, may be I missed something important...
Cheers,
Michel
Le 30/04/2024 à 13:24, Oliver Weinmann a écrit :
Hi,
I also have two domains. Default and one LDAP Active Directory. And had no issue. But if I recall correctly, I upgraded from yoga to zed and not directly to antelope.
Could it be related to the deprecation of the _member_ role?
Cheers, Oliver
Von meinem iPhone gesendet
Am 30.04.2024 um 11:37 schrieb Franck VEDEL franck.vedel@univ-grenoble-alpes.fr:
Hi.
With Antelope and 2 domains, same error. With Antelope and only one domain, no error. Maybe the same for you ?
Franck VEDEL
Le 30 avr. 2024 à 10:30, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :
Hi,
I am still stucked with this problem... Any other suggestion?
Cheers,
Michel
Le 29/04/2024 à 13:08, Michel Jouvin a écrit :
Hi Olivier,
Yes I have it.
Cheers,
Michel
Le 29/04/2024 à 12:45, Oliver Weinmann a écrit :
Hi Michel,
Do you have the following in your magnum.conf:
cat /etc/kolla/config/magnum.conf [trust] cluster_user_trust = True
Cheers, Oliver
Von meinem iPhone gesendet
> Am 29.04.2024 um 12:33 schrieb Michel Jouvin > michel.jouvin@ijclab.in2p3.fr: > > Hi, > > In a cloud where all services have been upgraded from Yoga to > Antelope, I tried to upgrade Magnum but after the (successful) > upgrade, Magnum reports "keystone client errors" when trying to > retrieve the status of existing clusters or to create new ones > (deletion of existing ones seem to work). > > The typical error is: > > ---- > > 2024-04-29 11:57:39.733 72464 WARNING magnum.service.periodic > [-] Skip pulling data from cluster > 2e8d6ee8-01cb-42d4-9b6d-0cf242cd4afa due to error: unexpected > keystone clie > nt error occurred: Forbidden (HTTP 403): > magnum.common.exception.AuthorizationFailure: unexpected > keystone client error occurred: Forbidden (HTTP 403) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > Traceback (most recent call last): > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/magnum/common/exception.py", > line 57, in wrapped > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > return func(*args, **kw) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/magnum/common/clients.py", > line 166, in barbican > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > self._barbican = barbicanclient.Client(session=session, > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/barbicanclient/client.py", > line 186, in Client > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > return client_class(session=session, *args, **kwargs) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", > line 47, in __init__ > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > normalized_microversion = self._get_max_supported_version( > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", > line 88, in _get_max_supported_version > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > min_ver, max_ver = self._get_min_max_server_supported_microversion( > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", > line 119, in _get_min_max_server_supporte > d_microversion > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > return self._get_min_max_version(session, endpoint, '1.1') > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/barbicanclient/v1/client.py", > line 125, in _get_min_max_version > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp > = discover.get_version_data( > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/keystoneauth1/discover.py", > line 107, in get_version_data > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic resp > = session.get(url, headers=headers, authenticated=authenticated) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", > line 1141, in get > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > return self.request(url, 'GET', **kwargs) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic File > "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", > line 986, in request > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > raise exceptions.from_response(resp, method, url) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > keystoneauth1.exceptions.http.Forbidden: Forbidden (HTTP 403) > 2024-04-29 11:57:39.733 72464 ERROR magnum.service.periodic > --- > > Did I miss something in the upgrade procedure? > > Thanks in advance for any help. Best regards, > > Michel >
On 1/5/2024 2:47 am, Michel Jouvin wrote:
I had a new look at the problem and found nothing to help as far as I can say... The keystone client error mentions a 403 status but I cannot find any trace of a matching request on the keystone server. All the requests from the Magnum/Heat server are completing with a 200 status according to the keystone (httpd) server logs...
Are there any errors in Barbican? The trace shows barbicanclient.
- Jake
Hi Jake,
Thanks for your answer, you got it! I didn't pay attention to the traceback and overlooked that the error was happing in Barbican client... The error was a Barbican httpd misconfiguration after the upgrade (a Python path not updated for the new Python version, 3.9). httpd log file was explict about it.
Cheers,
Michel
Le 01/05/2024 à 02:28, Jake Yip a écrit :
On 1/5/2024 2:47 am, Michel Jouvin wrote:
I had a new look at the problem and found nothing to help as far as I can say... The keystone client error mentions a 403 status but I cannot find any trace of a matching request on the keystone server. All the requests from the Magnum/Heat server are completing with a 200 status according to the keystone (httpd) server logs...
Are there any errors in Barbican? The trace shows barbicanclient.
- Jake
participants (4)
-
Franck VEDEL
-
Jake Yip
-
Michel Jouvin
-
Oliver Weinmann