April 2026 - openstack-discuss - lists.openstack.org

[barbican][oslo][nova][glance][cinder] cursive library status
by Brian Rosmaita 26 May '26

26 May '26

Hello Barbican team, Apologies for not including barbican in the previous thread on this topic: http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019430… The situation is that cursive is used by Nova, Glance, and Cinder and we'd like to move it out of the 'x' namespace into openstack governance. The question is then what team would oversee it. It seems like a good fit for Oslo, and the Oslo team seems OK with that, but since barbican-core is currently included in cursive-core, it make sense to give the Barbican team first dibs. From the consuming teams' side, I don't think we have a preference as long as it's clear who we need to bother about approvals if a bugfix is posted for review. Thus my ask is that the Barbican team indicate whether they'd like to move cursive to the 'openstack' namespace under their governance, or whether they'd prefer Oslo to oversee the library. Thank you! brian

6 6

[ops][nova] Performance problems in nova-api after Caracal → Epoxy update (and nova validators)
by federica fanzago 25 May '26

25 May '26

Dear all, summarizing the problem as much as possible, we have seen serious (blocking) performance problems after a Caracal → Epoxy update (on almalinux9 using RDO) that we were able to address only disabling the response_body_schema and query_schema nova validators. In more detail: to prepare the Caracal → Epoxy update of our production cloud, we have tried such update on a small testbed, composed by two controllers and four compute nodes.Please note that this testbed was initially installed using kilo, and then this cloud was regularly updated. There is a haproxy cluster in front of these 2 controllers, providing load balancing and also SSL terminations. The haproxy cluster uses OPTIONS nova webAPI for polling the service. As we already reported in this mailing list, after having updated the two controllers and one compute node everything seems ok, but the cpu usage of the node started increasing, as shown in the following Ganglia monitor dashboard: The processes consuming a lot of CPU are nova-api: after a couple of days they use 100% cpu of their thread (we are using 4 nova-api workers and the controller nodes have 8 cores), and everything becomes basically unusable. In particular we can see the problem trying to create a new VM: it takes a lot of time showing the list of available flavors (and often the operation fails). It looks like the checks done by the haproxy are enough to trigger the problem. This is the relevant section for nova-api in the haproxy conf file: frontend nova-api-public_infn bind <public-ip>:8774 ssl no-sslv3 crt /etc/cert/cloud-areapd_test_infn_chain_key.pem mode http option httpclose option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } default_backend nova-api-public frontend nova-api-public_unipd Bind <public-ip2>:8774 ssl no-sslv3 crt /etc/cert/cloud-areapd_test_unipd_chain_key.pem mode http option httpclose option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } default_backend nova-api-public backend nova-api-public mode http balance source option httpchk server controller-01.cloud.pd.infn.it 192.168.60.42:8774 check inter 2000 rise 2 fall 3 server controller-02.cloud.pd.infn.it 192.168.60.45:8774 check inter 2000 rise 2 fall 3 Attaching using strace to a nova-api process, we noticed that the calls to OPTIONS or GET (version) webAPIs trigger some “cryptographic calls” that we didn’t see in Caracal, e.g.: getrandom("\xdb\x08\x41\x11\x9b\x4a\x66\xc9\x14\x6d\x7d\xa2\xb3\xad\xf1\x79", 16, 0) = 16 getrandom("\xa2\x09\xbe\xcc\xa4\x60\x2a\xed\x21\xe1\x01\x1f\x21\xe8\x8c\x4c", 16, 0) = 16 getrandom("\x6f\xd0\x7b\xcf\x7c\xbb\x5f\x43\x78\xf8\xb8\xc6\x72\x45\xe8\x63", 16, 0) = 16 getrandom("\xc1\xc5\x58\x37\xaa\x0b\x85\x90\x61\xf9\x7e\xce\x42\x85\x1c\x9f", 16, 0) = 16 We presumed that the PRNG is somehow invoked by the schema validators. We then tried to disable the validation.response_body_schema(setting response_validation=ignore in nova.conf). This didn’t fix the problem (even if we noticed some improvement). We then disabled the *query_schema* (we had to patch the code) and this definitely helped: after some days we haven’t been seeing anymore the problem and the cpu usage doesn’t increase, as shown in the following picture: Please note that we are running AlmaLinux9 and we are using the latest rpm version from the RDO distribution. That means for nova we are using the version 31.0.0, which is not the latest version (since the packages in the RDO repo haven’t been updated after the initial epoxy release) but looking in the release note we don't find anything related to a problem with this symptom fixed in newer versions). We were told about a problem with similar symptoms (https://bugs.launchpad.net/nova/+bug/2121607) but we are not using those versions of python-jsonschema/python-attrs. Has anyone experienced similar problems ? Can you see/guess where the problem could be ? E.g. do you see problems in the haproxy configuration ? Are there known issues in these 2 nova validators ? How safe is disabling them ? Is it possible to disable the query_schema validator in a cleaner way than modifying the code ? Any help will be really appreciated ! Thanks a lot, cheers, Federica -- Federica Fanzago INFN Sezione di Padova Via Marzolo, 8 35131 Padova - Italy Tel: +39 049.967.7367 --

2 2

[kolla] kolla-core and kolla-reviewers proposed changes
by Michał Nasiadka 12 May '26

12 May '26

Hello, Kolla cores had a discussion about the current state of the core team, and we propose following changes. 1. Move an existing and long time Kolla-core to emeritus status: * Christian Berendt (cberendt) Christian has been a core reviewer for a long time, but sadly now he’s not that active anymore (no activity for the past year) - so we propose to move him to Emeritus status. This just means that while he may be removed from the team, if at some future point he returns to the Kolla community we may expedite his return to the core group. 2. Move an existing, but with minimum activity kolla-core to kolla-reviewers group: * Michal Arbet (kevko) While we value his contributions and technical insight on the reviews - we want to ensure that the final merge step is handled by those with real-time visibility into the latest commits and dependencies. 3. Last, but not least - promote the inaugural member of kolla-reviewers group to kolla-core. He has used his +2 judiciously, has been very active and tracking development priorities and have earned the trust of the community to use that same judgement about which changes to approve: * Bertrand Lanson (blanson) Thanks everyone for your contributions to Kolla, current and future. I’ll wait a week for responses to that proposal and implement given no critical feedback. Michal

6 5

[policy][tc] RBAC global auditor (reader)
by Ghanshyam Maan 07 May '26

07 May '26

Hi All, We discussed the need for a global auditor (reader across all projects/system-level) back in the Vancouver 2023 session, and all agreed that this is a good idea. For ironic and keystone, the solution is a system-level reader; for all other services, we have a reader scoped to the project only, with no reader that can read across projects. Cardoe brought up this use case today in the TC IRC channel, which was a reminder to me that we agreed on the idea but had not progressed on it. I would like to call this out on ML and ask for a volunteer who can start the work, which includes the spec in Keystone (example manager role) and working items mentioned in 'Work Items' [1]. I will not be able to help in implementation, but happy to help in review. [1] https://review.opendev.org/c/openstack/keystone-specs/+/818603/5/specs/keys… -gmaan

9 14

Reviewing Oslo core group memberships.
by Daniel Mats Niklas Bengtsson 07 May '26

07 May '26

Hi, I was thinking it might be a good time to review the oslo core groups memberships. Over time, some members may have become inactive, and doing a periodic cleanup could help ensure that the groups accurately reflect the current active maintainers. Would you agree with doing a review of the current memberships across the oslo projects? If there is consensus, I can help prepare a proposal listing potentially inactive members for discussion before making any changes. Thanks!

3 2

[stackalytics] no "neutron" module for Commits
by Ihar Hrachyshka 07 May '26

07 May '26

Hi, I noticed that "neutron" is not an option in the Module list when Commits are selected, and no neutron commits are registered since ~Oct 2023. On the other hand, other Neutron program repos (e.g. neutron-lib and neutron-tempest-plugin) are available in the list and do register commits. Also, when Reviews are selected, neutron module is back as an option, and review stats are correctly updated. I checked governance yaml and it doesn't strike me as incorrect: https://opendev.org/openstack/governance/src/commit/5f93b99a3c90b452f6c7368… Any ideas what may have happened? Thanks, Ihar

5 10

[neutron] tap-as-a-service-core group updates
by Brian Haley 06 May '26

06 May '26

Hi Neutrinos, As discussed at our PTG meeting, I will be adding Miro Tomaska to the tap-as-a-service-core group - he as made a number of contributions to Neutron over the years and has agreed to take a more active role in Tap-as-a-service maintenance going forward. Welcome Miro and good luck! I have also taken this time to clean-up the group and remove some members who are no longer active - Anil Rao, Kazuhiro Suzuki, Soichi Shigeta, Yamamoto Takashi and Vinay Yadhav. Thank you for all your previous contributions! -Brian, Neutron PTL

2 1

[cinder][nova][ceph] Cinder encryption happens at QEMU layer, not librbd — significant overhead, seeking direction
by Ravi Sasi Tilak 06 May '26

06 May '26

Hi all, We're running OpenStack with Ceph RBD as the Cinder storage backend and need encryption at rest with per-tenant key isolation. We've evaluated two approaches: 1. Cinder volume encryption Per-volume keys stored in OpenBao via Castellan. Cinder formats the RBD image with a LUKS header and QEMU handles decryption at attach time. The problem is that encryption always happens at the QEMU layer — Nova's libvirt driver doesn't set the engine to librbd, so encryption/decryption never happens inside librbd where it would be most efficient. It stays at the QEMU block layer, adding significant overhead. We're seeing 60-80% IOPS reduction in our testing. 2. Ceph librbd-native LUKS encryption Encryption happens inside librbd itself — client-side, per-image keys, data encrypted before it leaves the compute node. This is architecturally the right place for Ceph to do encryption. But there's no integration with external key managers — rbd_encryption_load() requires the passphrase directly from the caller, making it impractical for multi-tenant environments that need centralized key management. The gap: - Cinder encryption: per-tenant keys ✓, KMS integration ✓, performance ✗ (encryption at QEMU layer, not librbd) - librbd-native: performance ✓, per-image keys ✓, KMS integration ✗ What's the community's recommended approach for Ceph + OpenStack deployments that need per-tenant encryption keys with acceptable performance? Are we missing something, or is this a known gap? Happy to contribute if there's a path forward. Thanks, Ravi Sasi Tilak

5 4

Instance OS Disk Corruption on host failure
by Joel McLean 05 May '26

05 May '26

Hello Stackers, The Context: We've seen a few instances of disk corruption in the OS of instances where the underlying host had to be evacuated due to memory pressure causing the host to grind to a halt. The platform is Kolla Ansible 2024.1 (Caracal) on Ubuntu 22 hypervisors, with a very conventional configuration (nova-libvirt, neutron ml2-ovn). Notable is that all of the storage is delivered via Cinder using Ceph (Reef 18) RBD volume-backed storage on NVMe. The case in question we see the nova-evacuation take place with no problems, and in some cases the instances boot up with problems, complaining of symptoms consistent with disk corruption. We have seen this on a mix of NTFS, XFS, and EXT4 file systems. Using the windows as an example, services like chkdsk and dism are typically capable of a repair, although the same goes for ext4 and zfs own repair utilities. Typically, we expect to see the corruption occur on instances that are more likely to have a busy disk. We are able to replicate the fault by testing with SQL servers, for example. Given that the OS is generally capable of recovering from the corruption it's not a critical issue, as it can be fixed on a case-by-case basis, just something that we're checking with every migration or evacuation to make sure there are not any compounding problems. By way of comparison, we never saw this severity of disk corruption when a VMware host with VSAN storage would go offline unexpectedly. The Question What kinds of things could we be looking at to try and reduce the impact that a nova-evacuate might have on disk corruption? Or, Is this potentially more related to Ceph (which we've had no problems with, other than technically this) which could be resolved with Ceph tuning? Keen to hear your experiences. Kind Regards, Joel McLean Cyber Security and Product Development Manager Australia's First Tier IV Data Centre https://www.micron21.com/ 1300 769 972<tel:1300%20769%20972> 03 9751 7618<tel:03%209751%207618> 0407 888 429<tel:0407%20888%20429> joel.mclean(a)micron21.com<mailto:joel.mclean@micron21.com> <http://www.micron21.com/> [cid:image001.png@01D706E7.12E37EB0] Follow us on Twitter<https://twitter.com/micron21> and https://m21status.com<https://m21status.com/> for important service and system updates. This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

3 6

[all][security][ptls] Coresec teams need pruning
by Goutham Pacha Ravi 04 May '26

04 May '26

Hello Stackers, As you heard at the OpenInfra virtual PTG last week. We're seeing an uptick in the number of private security bugs that have been reported against OpenStack projects. Private bugs are handled by OpenStack's Vulnerability Management Team in conjunction with project "core security contacts". A majority of OpenStack projects track bugs on Launchpad. We use Launchpad's features to create "core security" groups there. Maintaining project core-sec teams with good hygiene is a priority. Embargoed data should only be accessible to people who are actively maintaining the project. Coresec membership should be a subset of core reviewers that are maintained on Gerrit. If someone has stepped back from the project, they should be removed from the coresec team promptly. PTLs, security liaisons: please treat this as a regular part of your role, not a one-time cleanup. Here's what I found that needs action today: oslo-coresec (8 members) ~~~~~~~~~~~~~~~~~~~~~~~~ bnemec (Ben Nemec), moguimar (Moises Guimaraes), sean-mcginnis (Sean McGinnis), and vbala (Vipin Balachandran) are no longer active in Oslo. That's half the team. damani42 (Daniel Bengtsson) recently raised the question of reviewing Oslo core groups [2]. The coresec team needs the same treatment. Please drop folks, and add anyone that's active and capable to help. designate-coresec (7 members) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ grahamhayes (Graham Hayes), kiall (Kiall Mac Innes), nicolasbock (Nicolas Bock), and timsim (Tim Simmons) have been inactive in Designate for a long time. johnsom (Michael Johnson) may have moved on too. Please clean up aggressively. That leaves j-harbott (Dr. Jens Harbott) and eandersson (Erik Andersson). Do we need to add more people here? swift-coresec (9 members) ~~~~~~~~~~~~~~~~~~~~~~~~~~ Quite large for a coresec team. notmyname (John Dickinson) may not be paying attention here anymore? alistair-coles (Alistair Coles) and thiagodasilva (Thiago da Silva) appear inactive; i'm not sure. Nine members is too many for a security review group where quick, focused response matters. I've tagged folks on a bug but not received responses. This could happen if everyone thinks someone else will look? nova-coresec (5 members) ~~~~~~~~~~~~~~~~~~~~~~~~ mikal (Michael Still) and johngarbutt (JohnGarbutt) may elect to be dropped? o-tony (Tony Breeds) seems less active too. Can the nova PTL be added here? If you need help to anoint administrators, please let me know. heat-coresec (6 members) ~~~~~~~~~~~~~~~~~~~~~~~~ Should this team be completely rebuilt? asalkeld (Angus Salkeld), jasondunsmore (Jason Dunsmore), skraynev (Sergey Kraynev), shardy (Steven Hardy), zaneb (Zane Bitter) and steve-stevebaker (Steve Baker) all seem to be either inactive, or not working on heat proper currently. horizon-coresec (6 members) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ amotoki (Akihiro Motoki), e0ne (Ivan Kolodyazhny), and vishalmanchanda (Vishal Manchanda) may need to be dropped? Anyone else to add in their place? In general, PTLs and DPL Security Liaisons, please: 1. Review your project's *-coresec team on Launchpad. 2. Remove anyone who is not an active core reviewer for your project. 3. If you're unsure about someone, ask them directly. No response in a week means they should be removed. 4. Verify your "coresec" team is owned by ~openstack-admins. If not, transfer it. 5. If your project doesn't have a coresec team and should, you should create one. 6. Make this a regular check every cycle. Please let us know on this list or on OFTC's #openstack-security if you have any concerns with any of this. Thank you, Goutham Pacha Ravi (gouthamr) OpenStack Vulnerability Management Team https://security.openstack.org/vmt-process.html

5 5