December 2023 - openstack-discuss

RE: openstack-discuss Digest, Vol 62, Issue 44 Neutron - Issue Urgent
by Asma Naz Shariq 14 Dec '23

14 Dec '23

Hi Openstack Neutron Team, I have deployed Multinode cluster of Openstack through kolla-ansible zed release. The problem I am facing is instances created cannot pick any Ip addresses( Rather than loop back ip or ipv6 addresses) from the assigned networks which I have created provider and self-service flat network type. Hence, I may not be able to ping and ssh instance from the controller node and these instances won’t be able to connect to the internet. Can you guide me how to resolve these issues related to network connectivity so I may able to send you the required files to troubleshoot easily? Thanks -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Thursday, December 14, 2023 12:26 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 62, Issue 44 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. What are the correct auth caps for Ceph RBD clients of Cinder / Glance / Nova (Christian Rohmann) 2. Re: What are the correct auth caps for Ceph RBD clients of Cinder / Glance / Nova (Jonathan Rosser) 3. Re: [tripleo][ironic][wallaby] Partition Table: loop (Alexey Kashavkin) 4. Re: [all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries (Jay Faulkner) ---------------------------------------------------------------------- Message: 1 Date: Wed, 13 Dec 2023 18:00:46 +0100 From: Christian Rohmann <christian.rohmann(a)inovex.de> Subject: What are the correct auth caps for Ceph RBD clients of Cinder / Glance / Nova To: openstack-discuss <openstack-discuss(a)lists.openstack.org> Message-ID: <a65d9057-67a2-4ab0-a2b6-fe327d185f06(a)inovex.de> Content-Type: text/plain; charset=UTF-8; format=flowed Hey openstack-discuss, I am a little confused about correct and required Ceph authx permissions for the RBD clients in Cinder, Glance and also Nova: When Glance is requested to delete an image it will check if this image has depended children, see https://opendev.org/openstack/glance_store/src/commit/6f5011d1f05c99894fb8b…. The children of Glance images usually are (Cinder) volumes, which therefore live in a different RBD pool "volumes". But if such children do exist a 500 error is thrown by Glance API. There also is an bug about this issue on Launchpad [3]. Manually using the RBD client shows the same error: > # rbd -n client.glance -k /etc/ceph/ceph.client.glance.keyring -p > images children $IMAGE_ID > > 2023-12-13T16:51:48.131+0000 7f198cf4e640 -1 > librbd::image::OpenRequest: failed to retrieve name: (1) Operation not > permitted > 2023-12-13T16:51:48.131+0000 7f198d74f640 -1 librbd::ImageState: > 0x5639fdd5af60 failed to open image: (1) Operation not permitted > rbd: listing children failed: (1) Operation not permitted > 2023-12-13T16:51:48.131+0000 7f1990c474c0 -1 librbd::api::Image: > list_descendants: failed to open descendant b7078ed7ace50d from pool > instances:(1) Operation not permitted So it's a permission error. Following either the documentation of Glance [1] or Ceph [2] on configuring the ceph auth caps there is no mention of granting anything towards the volume pool to Glance. So this is what I currently have configured: > client.cinder > key: REACTED > caps: [mgr] profile rbd pool=volumes, profile rbd-read-only > pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=volumes, profile rbd-read-only > pool=images > > client.glance > key: REACTED > caps: [mgr] profile rbd pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=images > > client.nova > key: REACTED > caps: [mgr] profile rbd pool=instances, profile rbd > pool=images > caps: [mon] profile rbd > caps: [osd] profile rbd pool=instances, profile rbd > pool=images > When granting the glance client e.g. "rbd-read-only" to the volumes pool via: > > # ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd > pool=images, profile rbd-read-only pool=volumes' mgr 'profile rbd > pool=images, profile rbd-read-only pool=volumes' > the error is gone. I am wondering through if this is really just a documentation bug (at OpenStack AND Ceph equally) and if Glance really needs read-only on the whole volumes pool or if there is some other capability that covers asking for child images. All in all I am simply wondering what the correct and least-privilege ceph auth caps for the RBD clients in Cinder, Glance and Nova would look like. Thanks Christian [1] https://docs.openstack.org/glance/latest/configuration/configuring.html#con… [2] https://docs.ceph.com/en/latest/rbd/rbd-openstack/#setup-ceph-client-authen… [3] https://bugs.launchpad.net/glance/+bug/2045158 ------------------------------ Message: 2 Date: Wed, 13 Dec 2023 17:40:44 +0000 From: Jonathan Rosser <jonathan.rosser(a)rd.bbc.co.uk> Subject: Re: What are the correct auth caps for Ceph RBD clients of Cinder / Glance / Nova To: openstack-discuss(a)lists.openstack.org Message-ID: <fb421a48-b0f6-4a6c-8d72-5b2ed8532b52(a)rd.bbc.co.uk> Content-Type: text/plain; charset=UTF-8; format=flowed Hi Christain, If you dig through the various deployment tooling then you'll find things like https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory… Hope this is helpful, Jonathan. On 13/12/2023 17:00, Christian Rohmann wrote: > Hey openstack-discuss, > > I am a little confused about correct and required Ceph authx > permissions for the RBD clients in Cinder, Glance and also Nova: > > When Glance is requested to delete an image it will check if this > image has depended children, see > https://opendev.org/openstack/glance_store/src/commit/6f5011d1f05c99894fb8b…. > The children of Glance images usually are (Cinder) volumes, which > therefore live in a different RBD pool "volumes". But if such children > do exist a 500 error is thrown by Glance API. There also is an bug > about this issue on Launchpad [3]. > > Manually using the RBD client shows the same error: > >> # rbd -n client.glance -k /etc/ceph/ceph.client.glance.keyring -p >> images children $IMAGE_ID >> >> 2023-12-13T16:51:48.131+0000 7f198cf4e640 -1 >> librbd::image::OpenRequest: failed to retrieve name: (1) Operation >> not permitted >> 2023-12-13T16:51:48.131+0000 7f198d74f640 -1 librbd::ImageState: >> 0x5639fdd5af60 failed to open image: (1) Operation not permitted >> rbd: listing children failed: (1) Operation not permitted >> 2023-12-13T16:51:48.131+0000 7f1990c474c0 -1 librbd::api::Image: >> list_descendants: failed to open descendant b7078ed7ace50d from pool >> instances:(1) Operation not permitted > > So it's a permission error. Following either the documentation of > Glance [1] or Ceph [2] on configuring the ceph auth caps there is no > mention of granting anything towards the volume pool to Glance. > So this is what I currently have configured: > >> client.cinder >> key: REACTED >> caps: [mgr] profile rbd pool=volumes, profile rbd-read-only >> pool=images >> caps: [mon] profile rbd >> caps: [osd] profile rbd pool=volumes, profile rbd-read-only >> pool=images >> >> client.glance >> key: REACTED >> caps: [mgr] profile rbd pool=images >> caps: [mon] profile rbd >> caps: [osd] profile rbd pool=images >> >> client.nova >> key: REACTED >> caps: [mgr] profile rbd pool=instances, profile rbd >> pool=images >> caps: [mon] profile rbd >> caps: [osd] profile rbd pool=instances, profile rbd >> pool=images >> > > When granting the glance client e.g. "rbd-read-only" to the volumes > pool via: >> >> # ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd >> pool=images, profile rbd-read-only pool=volumes' mgr 'profile rbd >> pool=images, profile rbd-read-only pool=volumes' >> > the error is gone. > > > I am wondering through if this is really just a documentation bug (at > OpenStack AND Ceph equally) and if Glance really needs read-only on > the whole volumes pool or if there is some other capability that > covers asking for child images. > > > All in all I am simply wondering what the correct and least-privilege > ceph auth caps for the RBD clients in Cinder, Glance and Nova would > look like. > > > > Thanks > > > Christian > > > > > [1] > https://docs.openstack.org/glance/latest/configuration/configuring.htm > l#configuring-the-rbd-storage-backend > [2] > https://docs.ceph.com/en/latest/rbd/rbd-openstack/#setup-ceph-client-a > uthentication [3] https://bugs.launchpad.net/glance/+bug/2045158 > > > ------------------------------ Message: 3 Date: Thu, 14 Dec 2023 01:05:41 +0600 From: Alexey Kashavkin <akashavkin(a)gmail.com> Subject: Re: [tripleo][ironic][wallaby] Partition Table: loop To: Julia Kreger <juliaashleykreger(a)gmail.com> Cc: openstack-discuss(a)lists.openstack.org Message-ID: <0F672C26-BC74-4D16-92FC-292F1D2BF62F(a)gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_BE0F9200-A2C4-458F-B2C4-31E6043ACAA2" Greetings Julia, Thank you so much for your reply and clarification. > On 12 Dec 2023, at 20:22, Julia Kreger <juliaashleykreger(a)gmail.com> wrote: > > Greetings Alexey, > > That would do it. Whenever we do not see a kernel/ramdisk with an image file, we assume it is a whole disk image, and as a result, we don't partition, we only write out what we've been provided. This is rooted in the image convention storage with Glance, because you need a kernel/ramdisk to boot a partition, but you can "just boot" a whole disk image. > > The disk utilities, in this specific case when you look at the disk, it thinks it is looking at a partition image on a loopback due to the lack of the partition table. > > That being said, whole disk images are the preferred path given you have greater flexibility in the partition/structure/layout of the deployed host. > > Hope that helps explain and provides the extra context. > > -Julia > > > > On Tue, Dec 12, 2023 at 2:56 AM Alexey Kashavkin <akashavkin(a)gmail.com <mailto:akashavkin@gmail.com>> wrote: >> I realized what was wrong in my case. I have a partition image, and I added the ramdisk and kernel path to 'baremetal_deployment.yaml'. This helped.

1 0

Re: [tripleo][ironic][wallaby] Partition Table: loop
by Alexey Kashavkin 13 Dec '23

13 Dec '23

I realized what was wrong in my case. I have a partition image, and I added the ramdisk and kernel path to 'baremetal_deployment.yaml'. This helped.

2 2

[Reminder] OpenInfra Live: Large Scale Ops Deep Dive Ft: Samsung SDS Dec. 14 at 15UTC
by Kristin Barrientos 13 Dec '23

13 Dec '23

Hi Everyone, Tune in this week for another episode of OpenInfra Live, brought to you by the Large Scale SIG. Episode: In the "Large Scale Ops Deep Dive" series, a panel of OpenStack operators invites special guests to talk about their deployment and discuss their operations. For this episode, our guest will be Dan Paik from Samsung SDS. Date and time: Thursday, Dec 14, 15 UTC YouTube: https://www.youtube.com/watch?v=RI9SwY8VdCY LinkedIn:https://www.linkedin.com/events/7125892582139654144/comments/ WeChat: The recording will be posted on OpenInfra Foundation WeChat after the live stream Speakers: Dan Paik, Arnaud Morin, Felix Huettner, Thierry Carrez, Kristin Barrientos Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

openstack cinder client help
by garcetto 13 Dec '23

13 Dec '23

good afternoon, i am trying to use the openstackclient ( https://docs.openstack.org/newton/user-guide/common/cli-install-openstack-c…) but it seems than I cannot use it to see cinder volume snapshot, even the doc say it is possible ( https://docs.openstack.org/python-openstackclient/latest/cli/decoder.html#c… ) [image: image.png] it says " (my-virt-env) user@ostack:~$ openstack snapshot list openstack: 'snapshot list' is not an openstack command. See 'openstack --help'. Did you mean one of these? endpoint add project endpoint create endpoint delete endpoint group add project endpoint group create ... " what am i missing? thank you.

3 2

[keystone] Keystone integration with Keycloak
by Franciszek Przewoźny 13 Dec '23

13 Dec '23

Hi all, I'm having some troubles integrating Keystone with Keycloak - pulling username and email from Keycloak using OIDC to Keystone works fine, but pulling groups (list of groups) doesn't. Keycloak provides such data in userinfo (pulled from Google, mapped in Keycloak and added to user): "usergroups": [ "group-a", "group-b", ], Keystone seems to be getting group names, but incorrectly parsing (?) it to a string with commas (instead of semicolon as it's required for multiple group entries to work: https://docs.openstack.org/keystone/latest/admin/federation/mapping_combina…) 2023-12-13 08:22:13.399 1076680 DEBUG keystone.federation.utils [None req-e8b8519a-c639-450e-b4e7-ca2ea8a0f3bc - - - - - -] updating a direct mapping: ['group-a,group-b'] _verify_all_requirements /usr/lib/python3/dist-packages/keystone/federation/utils.py:867 And tries to map user to group with names group-a,group-b: 2023-12-13 09:42:54.914 1084357 DEBUG keystone.federation.utils [None req-df016b4f-1ab4-4e1a-bd9f-314d9434b302 - - - - - -] mapped_properties: {'user': {'name': 'xxxxxxxx', 'email': 'xxxxxxx(a)yyyy.zzz', 'domain': {'id': 'default'}, 'type': 'ephemeral'}, 'group_ids': [], 'group_names': [{'name': 'group-a,group-b', 'domain': {'id': 'default'}}], 'projects': []} process /usr/lib/python3/dist-packages/keystone/federation/utils.py:562 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application [None req-df016b4f-1ab4-4e1a-bd9f-314d9434b302 - - - - - -] 'method' object is not subscriptable: TypeError: 'method' object is not subscriptable 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application Traceback (most recent call last): 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application rv = self.dispatch_request() 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 467, in wrapper 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application resp = resource(*args, **kwargs) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/flask/views.py", line 83, in view 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application return self.dispatch_request(*args, **kwargs) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application resp = meth(*args, **kwargs) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/server/flask/common.py", line 1064, in wrapper 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application return f(*args, **kwargs) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 386, in get 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application return self._perform_auth(idp_id, protocol_id) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 380, in _perform_auth 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application token = authentication.federated_authenticate_for_token( 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 251, in federated_authenticate_for_token 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application return authenticate_for_token(auth) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 185, in authenticate_for_token 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application authenticate(auth_info, auth_context) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 152, in authenticate 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application resp = method.authenticate(auth_info.get_method_data(method_name)) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/auth/plugins/mapped.py", line 59, in authenticate 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application response_data = handle_unscoped_token(auth_payload, 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/auth/plugins/mapped.py", line 225, in handle_unscoped_token 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application mapped_properties, mapping_id = apply_mapping_filter( 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application File "/usr/lib/python3/dist-packages/keystone/auth/plugins/mapped.py", line 321, in apply_mapping_filter 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application utils.validate_mapped_group_ids(group_ids, mapping_id, identity_api) 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application TypeError: 'method' object is not subscriptable 2023-12-13 09:42:54.920 1084357 ERROR keystone.server.flask.application My current setup: Google (user and groups membership) OIDC -> Keycloak OIDC -> Keystone Mapping rules (without parameters related to group settings works fine): [ { "local": [ { "user": { "name": "{0}", "email": "{1}", "domain": { "id": "default" } } }, { "group": { "name": "{2}", "domain": { "id": "default" } } } ], "remote": [ { "type": "OIDC-preferred_username" }, { "type": "OIDC-email" }, { "type": "OIDC-usergroups" } ] } ] WSGI Apache: ## WSGI configuration WSGIApplicationGroup %{GLOBAL} WSGIDaemonProcess keystone display-name=keystone group=keystone processes=4 threads=1 user=keystone WSGIProcessGroup keystone WSGIScriptAlias / "/usr/lib/cgi-bin/keystone/keystone" WSGIPassAuthorization On LoadModule auth_openidc_module modules/mod_auth_openidc.so OIDCClaimPrefix "OIDC-" OIDCResponseType "id_token" OIDCScope "openid email profile" OIDCProviderMetadataURL "https://keycloak.yyyy.zzz/realms/master/.well-known/openid-configuration <https://keycloak.opera.software/realms/master/.well-known/openid-configurat…>" OIDCClientID "openstack" Any help appreciated! Thanks in advance, Franciszek Przewoźny

1 1

[all][tc][cinder] deprecating cinderlib
by Brian Rosmaita 13 Dec '23

13 Dec '23

As discussed at our midcycle meeting last week [0], the Cinder project team hereby announces the deprecation of cinderlib, a library that allows the direct usage of Cinder backend drivers without the cinder services. Cinderlib is a cycle-trailing release, and thus this announcement comes on the heels of the 2023.2 (Bobcat) release of cinderlib, which happened just last week. The Cinder project team will continue to support cinderlib in the stable Maintained branches for their normal lifecycles, as specified by the OpenStack Stable Branch Policy: - stable/2023.2: end of maintenance Spring 2025 - stable/2023.1: end of maintenance Fall 2024 - stable/zed: end of maintenance Spring 2024 The stable/victoria through stable/yoga branches are in Unmaintained status. If you have questions or concerns about the deprecation of cinderlib, please reply to this email or leave a comment on the governance patch: https://review.opendev.org/c/openstack/governance/+/903259 Thank you for your attention to this matter. [0] https://wiki.openstack.org/wiki/CinderCaracalMidCycleSummary

2 1

[openstack-ansible] December and January meetings schedule
by Neil Hanlon 13 Dec '23

13 Dec '23

Hello all, and happy holidays :) The OSA team will be skipping our regularly scheduled meetings on December 26th and January 2nd due to the Christmas/New Year festivities. We'll be meeting once more before the end of the year, next Tuesday, December 19th. Thank you, and I wish you all the best in 2024! Cheers, Neil

1 0

System configuration in OpenStack
by lostghost1＠yandex.ru 12 Dec '23

12 Dec '23

Hi all, Sorry if this has been asked already, but is it feasible to use OpenStack to configure running systems - including virtual instances, containers, and the physical nodes themselves. I know that conceptually OpenStack takes a hands-off approach - it deals with images in abstract and further configuration is delegated to tools such as Ansible. Same for bare metal nodes - you provision them with Ironic, flash an image to them, and that's where configuaration ends for OpenStack. At least this is how I understand it, may be wrong. I was thinking about introducing systems configuration as a first-class concept into OpenStack - that means integration with the package manager, writing of config files from templates, filesystem management in general. The most well thought-out "solution" that there is for systems configuration is Nix in my opinion. I could intergrate it into OpenStack, I could write my own package manager. I haven't decided yet, these are just ideas for a pet project. So if I were to do such a thing, how would I go about achieving it? Obviously I would need to write a Horizon console module for the web UI. But beyond that I'd also need integration with other components. What do you all think?

3 6

[publiccloud-sig] Reminder - next meeting December 13th - 0800 UTC
by Felix Kronlage-Dammers 12 Dec '23

12 Dec '23

Hi everyone, This Wednesday the next meeting of the public cloud sig is going to happen. Since we do every other meeting as a videosession, we’ll be meeting at 0800 UTC here: https://conf.scs.koeln:8443/OIF-public-cloud-sig We’re going to continue our discussion on user stories for the discoverability of flavors and alike. Please see https://etherpad.opendev.org/p/publiccloud-sig-meeting for further infos. See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG See you tomorrow! felix -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

Issuse with OVN DVR generate too much flow on compute node
by Sang Tran Quoc 12 Dec '23

12 Dec '23

Hi OpenStack Community, I've indicated very weird behavior about ovn flows when VM is associated with vrouter, I use one new compute node with only 1 VM for testing and result as below: * When VM attach to vRouter without external network: Only add new 2047 flows. * When vRouter attach to external network (provider-net2): It will generate 147k flows immediately ? * We've tested with VM direct attach to provider-net2 also generate 147k flows. [cid:image001.png@01DA2D02.780FB210] My environment is OpenStack Victoria version, neutron deployed with OVN, DVR, Distributed FIP mode, and provider-net2 using VLAN setup. The problems lead to the heavily process of sync flow which slows down the normal VM operations and packet drop a little bit while update large flow on the ovswitch. Could anyone has experienced this behavior before or have a solution config/patch neutron to address this? Any suggestions are highly appreciated. Thank and regards, Sang Tran

1 0