June 2024 - openstack-discuss - lists.openstack.org

[Horizon][Keystone] Multiple OIDC Providers and token deletion at logout
by MIchele Delli Veneri 06 Jun '24

06 Jun '24

Hi, We have configured our keystone and Horizon to work with multiple OIDC Providers and through API I can successfully authenticate with the two providers. When I authenticate through the Horizon dashboard, I can successfully authenticate with both providers if I use two separate browser sessions, If I use the same browser session, I log in with one provider, log out, and then log in with the other, given that the user token is stored, I get an authentication error. The reason for the error is that the token was issued by the first provider and thus it is not accepted as a valid by the second. Is it possible to configure Horizon to revoke/delete a user token at logout or provide session isolation without changes on the provider side? Thank you and best regards, Michele Delli Veneri

1 0

[Horizon][Keystone] Multiple OIDC Providers and token deletion at logout
by Michele Delli Veneri 06 Jun '24

06 Jun '24

Hi, We have configured our keystone and Horizon to work with multiple OIDC Providers and through API I can successfully authenticate with the two providers. When I authenticate through the Horizon dashboard, I can successfully authenticate with both providers if I use two separate browser sessions, If I use the same browser session, I log in with one provider, log out, and then log in with the other, given that the user token is stored, I get an authentication error. The reason for the error is that the token was issued by the first provider and thus it is not accepted as a valid by the second. Is it possible to configure Horizon to revoke/delete a user token at logout or provide session isolation without changes on the provider side? Thank you and best regards, Michele Delli Veneri

1 0

Re: [ops] help with manila
by Alvaro Soto 05 Jun '24

05 Jun '24

[ please hit reply all ] Hi! 1. Your cluster needs to be healthy for Manila to connect. Could you purge your Ceph cluster to free up space? 2. They are very open, but they are OK if they are just for testing. 3. Yes, many resources are out there, but I'll depend on what deployment tool you're using. Cheers!

1 0

Antelope Barbican/Keystone: failure to delete secrets (Keystone TrustNotFound)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I've been validating Yoga -> Antelope on our preprod cloud. It was working successfully but "suddenly" (i.e. without a kwown/identified change!) when deleting Magnum clusters, the deletion of Barbican secrets (after stack deletion) fail. In the Magnum log, I find the following error: ---- Failed to delete trust: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a4400250-e8d5-46ec-a28a-dcb97a682512) (traceback skipped for readability) 2024-05-29 10:43:09.750 2078 INFO magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17… from Barbican. 2024-05-29 10:43:09.750 2078 INFO barbicanclient.base [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Calculated Containers uuid ref: containers/edd8487a-6cf1-4f4c-9f53-173dfae7a8b2 2024-05-29 10:43:09.760 2078 ERROR barbicanclient.client [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-29 10:43:09.761 2078 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17…: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. ---- In Keystone log, I find with the same timestamp the following error that seems related to the one above: ---- 2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ---- It is not clear for me if the ERROR is releated to the 2 warnings as the request is not the same but it seems to happen every time this way. In Barbican logs, there is not much things related except a 401 status for the request corresponding ot the TrustNotFound. I'm looking for advices on how to troubleshoot this. In particular, when the trust should be created and whether I should find somewhere a matching error at the time of the trust creation. It seems the error is only affecting deletion but I may be wrong. for the record, when starting Magnum there is an error complaining about /etc/magnum/keystone_auth_default_policy.json being not found but ignored it so far. And in Keystone an error seems to occur at Magnum cluster creation time: --- 2024-05-29 10:17:17.530 1711 WARNING py.warnings [None req-f484cc35-3e54-4d7f-8688-e6442c90344f a22b1199c74f4a819a49f3ac037a5e44 1dd67113c8bb4ef0a31dfca53335ffbd - - defaul t default] /usr/lib/python3.9/site-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_services": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is re quired ---- Thanks in advance for any hint. Best regards, Michel

2 3

Antelope Magnum: creating a new cluster makes the/some others unhealthy
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I have been playing with Magnum (Antelope), creating a lot of clusters with the same template and varying the number of masters and/or nodes, having ~10 clusters started in the same project. I observed recently that cluster configuration that used to work well were not working any more (CREATE_FAILED during the master deployment in general). I have not found any evidence of the cause digging in various logs but I observed that with the current list of active clusters, if add a new one (even with a configuration as minimal as 1 master and 1 node), not only its creation fails but also the other (or at least) running clusters become unhealthy. If I delete the failed cluster, the other clusters return to the healthy state. It seems very reproducible (I did it more than 10 times) but I still don't see any message in the logs that could help to identify the cause. I've the feeling (but may be wrong) that it is related to either an insufficient project quota or an insufficient limit on one of the OpenStack service. Any idea on a possible cause or any advice on where to look for some information? Thanks in advance. Michel

2 6

Antelope: weird problem with member role after upgrade (Nova and Glance)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). For (bad) historical reasons, the role "member" was called "users" but it had no impact (I was surprised), despite we are using standard policies. We thought it may be a consequence of this and we renamed the role back to "member". It was not enough to fix the problem, even after restart memcached on all servers just in case. We thought that there was may be some caching done somewhere with the old role name and modified slightly the policy rules defining what is a member or read with: "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" It first works but the change was reverted by mistake and now it doesn't work anymore. I am really completely stuck, without any clue about what happen and on how to troubleshoot it. I googled a bit but was not able to find something looking similar... Any help would be greatly appreciated. Best regards, Michel -- Michel

3 2

[ops] help with manila
by Shubham Kumar Yadav 04 Jun '24

04 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem with "*Authorizing the driver to communicate with Ceph*" *1 -> *i have created a configmap with these confs and mounted it on manila-share pod [client.manila]client mount uid = 0client mount gid = 0log file = /opt/stack/logs/ceph-client.manila.logadmin socket = /opt/stack/status/stack/ceph-$name.$pid.asokkeyring = /etc/ceph/manila.keyring *2 -> *then i have created a secret with client.manila keyring and attached it to the manila-share pod at keyring = /etc/ceph/manila.keyring *3 -> *inside manila-share pod i see the logs 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 *4 -> *and in manila-api i see logs Mon Jun 3 09:41:18 2024 - *** uWSGI listen queue of socket "0.0.0.0:8786" (fd: 3) full !!! (101/100) *** Mon Jun 3 09:41:19 2024 - *** uWSGI listen queue of socket "0.0.0.0:8786" (fd: 3) full !!! (101/100) *** even when i have set manila: DEFAULT: wsgi_default_pool_size: 1000 Hope to hear from you soon with a solution Thank you!

2 1

Is it possible to rename a role?
by Michel Jouvin 04 Jun '24

04 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically with Glance and Nova, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). After some research, we found that it was linked to the fact that for (bad) historical reasons, the person who created the cloud 10 years ago renamed the role "member" to "users". It has been working without bad impact up to know. I was surprised of this absence of impact when we switched to scope tokens and enforced new rules. I realize that it is because several (all?) services up to Yoga was still supporting the old rules in addition to the new ones. On Cinder and Keystone for example, if using oslopolicy-policy-generator, it reports deprecated rules that are not defined in our configuration and I guess it is because old rules are not yet deleted on these services. Our first attempt to fix the problem has been to rename the users role (that every non-admin user has instead of the no-longer existing member role) back to its original name, member. Unfortunately it didn't solve the problem, even after restarting memcached everywhere (not sure it may have an impact) and the API services on Glance and Nova. We found a workaround modifying slightly the policy rules defining what is a member or reader with the following for Nova (and something similar for Glance, even if more rules have to be modified as there is no rule defining what is a member or reader): "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" *My question is:* why renaming the role was not enough? The fact that "openstack role set --name" exists suggests that it should work, isn't it? We can probably leave with the workound we found but it would make the configuration simpler if we could fix the role name and keep the standard rules... Thanks in advance for any suggestion on how to do it or to troubleshoot the fact it doesn't work... Best regards, Michel

1 0

Is it possible to rename a role?
by Michel Jouvin 04 Jun '24

04 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically with Glance and Nova, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). After some research, we found that it was linked to the fact that for (bad) historical reasons, the person who created the cloud 10 years ago renamed the role "member" to "users". It has been working without bad impact up to know. I was surprised of this absence of impact when we switched to scope tokens and enforced new rules. I realize that it is because several (all?) services up to Yoga was still supporting the old rules in addition to the new ones. On Cinder and Keystone for example, if using oslopolicy-policy-generator, it reports deprecated rules that are not defined in our configuration and I guess it is because old rules are not yet deleted on these services. Our first attempt to fix the problem has been to rename the users role (that every non-admin user has instead of the no-longer existing member role) back to its original name, member. Unfortunately it didn't solve the problem, even after restarting memcached everywhere (not sure it may have an impact) and the API services on Glance and Nova. We found a workaround modifying slightly the policy rules defining what is a member or reader with the following for Nova (and something similar for Glance, even if more rules have to be modified as there is no rule defining what is a member or reader): "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" My question is: why renaming the role was not enough? The fact that "openstack role set --name" exists suggests that it should work, isn't it? We can probably leave with the workound we found but it would make the configuration simpler if we could fix the role name and keep the standard rules... Thanks in advance for any suggestion on how to do it or to troubleshoot the fact it doesn't work... Best regards, Michel -- Michel

1 0

Nova live migration
by Gk Gk 04 Jun '24

04 Jun '24

Hi, I see this in the latest nova configuration: This option is deprecated for removal since 15.0.0. Its value may be silently ignored in the future. Reason: live_migration_uri is deprecated for removal in favor of two other options that allow to change live migration scheme and target URI: live_migration_scheme and live_migration_inbound_addr respectively. So, if I want to use live_migration_scheme, what are its values ? If I use the live_migration_scheme option, can I remove the live_migration_uri option ? Thanks

3 9