June 2024 - openstack-discuss - lists.openstack.org

Re: [ops] help with manila
by Alvaro Soto 06 Jun '24

06 Jun '24

[ please hit reply all ] Hi! 1. Your cluster needs to be healthy for Manila to connect. Could you purge your Ceph cluster to free up space? 2. They are very open, but they are OK if they are just for testing. 3. Yes, many resources are out there, but I'll depend on what deployment tool you're using. Cheers!

1 0

Antelope Barbican/Keystone: failure to delete secrets (Keystone TrustNotFound)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I've been validating Yoga -> Antelope on our preprod cloud. It was working successfully but "suddenly" (i.e. without a kwown/identified change!) when deleting Magnum clusters, the deletion of Barbican secrets (after stack deletion) fail. In the Magnum log, I find the following error: ---- Failed to delete trust: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a4400250-e8d5-46ec-a28a-dcb97a682512) (traceback skipped for readability) 2024-05-29 10:43:09.750 2078 INFO magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17… from Barbican. 2024-05-29 10:43:09.750 2078 INFO barbicanclient.base [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Calculated Containers uuid ref: containers/edd8487a-6cf1-4f4c-9f53-173dfae7a8b2 2024-05-29 10:43:09.760 2078 ERROR barbicanclient.client [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-29 10:43:09.761 2078 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17…: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. ---- In Keystone log, I find with the same timestamp the following error that seems related to the one above: ---- 2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ---- It is not clear for me if the ERROR is releated to the 2 warnings as the request is not the same but it seems to happen every time this way. In Barbican logs, there is not much things related except a 401 status for the request corresponding ot the TrustNotFound. I'm looking for advices on how to troubleshoot this. In particular, when the trust should be created and whether I should find somewhere a matching error at the time of the trust creation. It seems the error is only affecting deletion but I may be wrong. for the record, when starting Magnum there is an error complaining about /etc/magnum/keystone_auth_default_policy.json being not found but ignored it so far. And in Keystone an error seems to occur at Magnum cluster creation time: --- 2024-05-29 10:17:17.530 1711 WARNING py.warnings [None req-f484cc35-3e54-4d7f-8688-e6442c90344f a22b1199c74f4a819a49f3ac037a5e44 1dd67113c8bb4ef0a31dfca53335ffbd - - defaul t default] /usr/lib/python3.9/site-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_services": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is re quired ---- Thanks in advance for any hint. Best regards, Michel

2 3

Antelope Magnum: creating a new cluster makes the/some others unhealthy
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I have been playing with Magnum (Antelope), creating a lot of clusters with the same template and varying the number of masters and/or nodes, having ~10 clusters started in the same project. I observed recently that cluster configuration that used to work well were not working any more (CREATE_FAILED during the master deployment in general). I have not found any evidence of the cause digging in various logs but I observed that with the current list of active clusters, if add a new one (even with a configuration as minimal as 1 master and 1 node), not only its creation fails but also the other (or at least) running clusters become unhealthy. If I delete the failed cluster, the other clusters return to the healthy state. It seems very reproducible (I did it more than 10 times) but I still don't see any message in the logs that could help to identify the cause. I've the feeling (but may be wrong) that it is related to either an insufficient project quota or an insufficient limit on one of the OpenStack service. Any idea on a possible cause or any advice on where to look for some information? Thanks in advance. Michel

2 6

Antelope: weird problem with member role after upgrade (Nova and Glance)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). For (bad) historical reasons, the role "member" was called "users" but it had no impact (I was surprised), despite we are using standard policies. We thought it may be a consequence of this and we renamed the role back to "member". It was not enough to fix the problem, even after restart memcached on all servers just in case. We thought that there was may be some caching done somewhere with the old role name and modified slightly the policy rules defining what is a member or read with: "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" It first works but the change was reverted by mistake and now it doesn't work anymore. I am really completely stuck, without any clue about what happen and on how to troubleshoot it. I googled a bit but was not able to find something looking similar... Any help would be greatly appreciated. Best regards, Michel -- Michel

3 2

[ops] help with manila
by Shubham Kumar Yadav 05 Jun '24

05 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem with "*Authorizing the driver to communicate with Ceph*" *1 -> *i have created a configmap with these confs and mounted it on manila-share pod [client.manila]client mount uid = 0client mount gid = 0log file = /opt/stack/logs/ceph-client.manila.logadmin socket = /opt/stack/status/stack/ceph-$name.$pid.asokkeyring = /etc/ceph/manila.keyring *2 -> *then i have created a secret with client.manila keyring and attached it to the manila-share pod at keyring = /etc/ceph/manila.keyring *3 -> *inside manila-share pod i see the logs 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 *4 -> *and in manila-api i see logs Mon Jun 3 09:41:18 2024 - *** uWSGI listen queue of socket "0.0.0.0:8786" (fd: 3) full !!! (101/100) *** Mon Jun 3 09:41:19 2024 - *** uWSGI listen queue of socket "0.0.0.0:8786" (fd: 3) full !!! (101/100) *** even when i have set manila: DEFAULT: wsgi_default_pool_size: 1000 Hope to hear from you soon with a solution Thank you!

2 1

Is it possible to rename a role?
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically with Glance and Nova, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). After some research, we found that it was linked to the fact that for (bad) historical reasons, the person who created the cloud 10 years ago renamed the role "member" to "users". It has been working without bad impact up to know. I was surprised of this absence of impact when we switched to scope tokens and enforced new rules. I realize that it is because several (all?) services up to Yoga was still supporting the old rules in addition to the new ones. On Cinder and Keystone for example, if using oslopolicy-policy-generator, it reports deprecated rules that are not defined in our configuration and I guess it is because old rules are not yet deleted on these services. Our first attempt to fix the problem has been to rename the users role (that every non-admin user has instead of the no-longer existing member role) back to its original name, member. Unfortunately it didn't solve the problem, even after restarting memcached everywhere (not sure it may have an impact) and the API services on Glance and Nova. We found a workaround modifying slightly the policy rules defining what is a member or reader with the following for Nova (and something similar for Glance, even if more rules have to be modified as there is no rule defining what is a member or reader): "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" *My question is:* why renaming the role was not enough? The fact that "openstack role set --name" exists suggests that it should work, isn't it? We can probably leave with the workound we found but it would make the configuration simpler if we could fix the role name and keep the standard rules... Thanks in advance for any suggestion on how to do it or to troubleshoot the fact it doesn't work... Best regards, Michel

1 0

Is it possible to rename a role?
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically with Glance and Nova, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). After some research, we found that it was linked to the fact that for (bad) historical reasons, the person who created the cloud 10 years ago renamed the role "member" to "users". It has been working without bad impact up to know. I was surprised of this absence of impact when we switched to scope tokens and enforced new rules. I realize that it is because several (all?) services up to Yoga was still supporting the old rules in addition to the new ones. On Cinder and Keystone for example, if using oslopolicy-policy-generator, it reports deprecated rules that are not defined in our configuration and I guess it is because old rules are not yet deleted on these services. Our first attempt to fix the problem has been to rename the users role (that every non-admin user has instead of the no-longer existing member role) back to its original name, member. Unfortunately it didn't solve the problem, even after restarting memcached everywhere (not sure it may have an impact) and the API services on Glance and Nova. We found a workaround modifying slightly the policy rules defining what is a member or reader with the following for Nova (and something similar for Glance, even if more rules have to be modified as there is no rule defining what is a member or reader): "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" My question is: why renaming the role was not enough? The fact that "openstack role set --name" exists suggests that it should work, isn't it? We can probably leave with the workound we found but it would make the configuration simpler if we could fix the role name and keep the standard rules... Thanks in advance for any suggestion on how to do it or to troubleshoot the fact it doesn't work... Best regards, Michel -- Michel

1 0

Nova live migration
by Gk Gk 04 Jun '24

04 Jun '24

Hi, I see this in the latest nova configuration: This option is deprecated for removal since 15.0.0. Its value may be silently ignored in the future. Reason: live_migration_uri is deprecated for removal in favor of two other options that allow to change live migration scheme and target URI: live_migration_scheme and live_migration_inbound_addr respectively. So, if I want to use live_migration_scheme, what are its values ? If I use the live_migration_scheme option, can I remove the live_migration_uri option ? Thanks

3 9

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-17)
by Goutham Pacha Ravi 04 Jun '24

04 Jun '24

Hello Stackers, The 2024.2 release cycle is chugging along, and its last stop is 17 weeks away [1]. This week is the 2024.1 "cycle-trailing" release deadline for projects that follow this model. Typically, these are configuration management, packaging, and deployment tooling projects. In the past week, we undertook several governance activities that I'd like to summarize for you. We made a significant change to the "Distributed Leadership" model of project governance [2]. A project's governance will default to the PTL model during elections; if the project community would like to pursue the DPL model, they can do so by nominating liaisons for that release cycle. There's also going to be a TC liaison to each project following the DPL model. The TC liaison will follow the project's activity at regular intervals and interface with the TC. These changes will hopefully help signal project health concerns earlier. The TC also clarified that we'd adopt a "formal-vote" process for project team retirements. A formal vote would mean a third of the TC endorses a change through a "roll call" vote. We also cleaned up the emerging and inactive projects list [3]. Members of the TaCT SIG have started the arduous process of cleaning up PyPi, where we had a number of human maintainer accounts for OpenStack packages [4]. === Weekly Meeting === The OpenStack Technical Committee (TC) meeting on May 28, 2024, covered several topics [5]. Concerns were raised about the inactivity of projects like Freezer, Monasca, and Watcher. Discussions focused on adding TC members as temporary core reviewers to inactive projects to maintain them until new maintainers are found or to retire them if no interest is shown. A vote was held on whether the TC should be added to the Gerrit approving group for inactive projects, resulting in a split decision (4 yes, 3 no). For now, we decided to maintain the status quo. The revival of Watcher is being spearheaded by Dan Smith (dansmith) and aided by Ghanshyam Mann (gmann) and Slawek Kaplonski (slaweq). Our next weekly meeting will be held tomorrow (June 4, 2024) simultaneously on video (Zoom) and IRC. The meeting agenda and the details on how to join are in [6]. I hope that you're able to join us! === Governance Proposals === ==== Merged ==== - Remove retired project from Inactive project list | https://review.opendev.org/c/openstack/governance/+/920146 - Mark migrate-ci-jobs-to-ubuntu-jammy goal completed | https://review.opendev.org/c/openstack/governance/+/920143 - Add DPL model & liaison reset policy | https://review.opendev.org/c/openstack/governance/+/916833 - Clarify the project retirement vs project's repository retirement | https://review.opendev.org/c/openstack/governance/+/919984 - Simplifying house rule for project-updates | https://review.opendev.org/c/openstack/governance/+/920318 ==== Open for review/discussion ==== - Appoint ChunYang Wu as PTL of Trove | https://review.opendev.org/c/openstack/governance/+/920462 - Migrate eventlet usages to asyncio | https://review.opendev.org/c/openstack/governance/+/902585 - Update to include docs and miscellaneous repos for AC status | https://review.opendev.org/c/openstack/governance/+/915021 The OpenStack Technical Committee serves you! We're always willing to chat with you and would love feedback: === How to contact the TC === You can reach out to the TC in multiple ways: 1. Email: you can send the email with the tag [tc] on this email list. 2. Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel. 3. Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [6]. Finally, if you got this far, I hope you found this update helpful. I'd appreciate it if you had any feedback regarding the format, content, frequency, or timing of this weekly email. Please feel free to email me off-list if you're more comfortable doing so. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://releases.openstack.org/dalmatian/schedule.html [2] https://governance.openstack.org/tc/reference/distributed-project-leadershi… [3] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [4] https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L52 [5] https://meetings.opendev.org/meetings/tc/2024/tc.2024-05-28-18.00.log.html [6] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[all] SQLAlchemy minimum requirement
by Pierre Riteau 03 Jun '24

03 Jun '24

Hello everyone, Historically, many projects were using the following requirement for SQLAlchemy: SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 These banned versions were present in global-requirements.txt until earlier this month [1]. The remaining projects with the above requirement must change it to fix their requirements-check job (ERROR: Requirement for package SQLAlchemy excludes a version not excluded in the global list). While fixing the requirement for blazar, I realised that OpenStack projects are using a wide range of minimum requirements. For example, networking-sfc had already bumped the minimum to SQLAlchemy 2.0 [2], although this is being reverted [3]. While all projects must be compatible with the upper constraint (currently 2.0.30), should we agree on a maximum version for the lower bound? The 2024.2 release targets Ubuntu Jammy, which includes python3-sqlalchemy 1.4.31, and Debian bookworm, which includes python3-sqlalchemy 1.4.46. Thanks, Pierre Riteau (priteau) [1] https://review.opendev.org/c/openstack/requirements/+/879743 [2] https://review.opendev.org/c/openstack/networking-sfc/+/920046 [3] https://review.opendev.org/c/openstack/networking-sfc/+/920868

2 1