July 2025 - openstack-discuss - lists.openstack.org

[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
by Renu Rani 30 Jul '25

30 Jul '25

Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5d…] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌラニ) Mobile: +91-9311888962 E-mail: renu.rani(a)india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.

5 10

[neutron] Is NVIDIA ASAP2 offloading with OpenStack just a myph?
by Vasyl Saienko 30 Jul '25

30 Jul '25

Hello OpenStack community, I'm trying to set up OpenStack with Nvidia ASAP2 hardware offloading in switchdev mode according to OpenStack guidelines <https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html> and Nvidia guidelines <https://docs.nvidia.com/doca/sdk/ovs-kernel+hardware+offloads/index.html> on the ConnectX6DX. I've tried different network types VLAN and VXLAN but offloading for none of them works. From an OpenStack standpoint everything is configured according to documentation, representor device is created and plugged in the br-int bridge (eth14) physical. PF enp33s0f0np0 is set to switchdev mode ``` devlink dev eswitch show pci/0000:21:00.0 pci/0000:21:00.0: mode switchdev inline-mode none encap-mode basic ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl show 12e00cdb-6520-4666-908f-57f59e8b9bd4 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-ex Interface br-ex type: internal Port pr-floating Interface pr-floating Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port int-br-tenant Interface int-br-tenant type: patch options: {peer=phy-br-tenant} Port tap58ac2b17-fb tag: 1 Interface tap58ac2b17-fb Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port eth14 tag: 1 Interface eth14 Bridge br-tenant Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tenant Interface br-tenant type: internal Port phy-br-tenant Interface phy-br-tenant type: patch options: {peer=int-br-tenant} Port enp33s0f0np0 Interface enp33s0f0np0 Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} ovs_version: "3.0.0-0056-25.04-based-3.3.5" ``` But hardware offloading is not happening, tcpdump on eth14 still shows all packets, and no TC rules are created. OpenStack uses complex openvswitch rules to setup virtual networking, in this scenario vlan tagging is done by the following rule ``` cookie=0x45b91976d5c4f175, duration=233.443s, table=60, n_packets=314, n_bytes=35574, priority=100,in_port=eth14 actions=set_field:0x11->reg5,set_field:0x1->reg6,resubmit(,73) ``` If we simplify openflow rules and align them with Nvidia guidelines for openvswitch hardware offloading by plugging representor device directly into physnet bridge, and assign vlan tag as port attribute hardware offloading starts working. ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl del-port eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl add-port br-tenant eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl set port eth14 tag=1443 ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# for i in $(ip link | awk '{print $2}' |grep -e enp -e ens |grep -v vf$); do tc filter show dev $i ingress; done filter protocol 802.1Q pref 6 flower chain 0 filter protocol 802.1Q pref 6 flower chain 0 handle 0x1 vlan_id 1443 vlan_prio 0 vlan_ethtype arp dst_mac fa:16:3e:36:1b:90 src_mac fa:16:3e:39:90:d8 eth_type arp in_hw in_hw_count 1 action order 1: vlan pop pipe index 1 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth14) stolen index 1 ref 1 bind 1 cookie 4b5b5a701f4e3fee7ac1c198037522e3 no_percpu used_hw_stats delayed ``` Similar issue occurs when using the VXLAN network. OpenStack setup VXLAN tunnel like, where tunnel_id is inserted dynamically via openflow rules (in_key=flow) ``` Port vxlan-0a0c0004 Interface vxlan-0a0c0004 type: vxlan options: {df_default="true", dst_port="4790", egress_pkt_mark="0", in_key=flow, local_ip="10.12.0.9", out_key=flow, remote_ip="10.12.0.4"} ``` In this configuration hardware offloading is not happening neither. But when setting up vxlan according to nvidia guidelines (no complex rules are involved) hardware offloading starts working (tcpdump for representor device shows only first packet and tc rules are being created). ``` Port vxlan3 Interface vxlan3 type: vxlan options: {key="60987", local_ip="10.12.0.9", remote_ip="10.12.0.4"} ``` ``` Cannot find device "enp33s0f0npf0vf" Cannot find device "veth68d427d@if1" filter protocol ip pref 2 flower chain 0 filter protocol ip pref 2 flower chain 0 handle 0x1 dst_mac fa:16:3e:65:0c:6d src_mac 52:5a:d0:9e:8a:32 eth_type ipv4 enc_dst_ip 10.12.0.9 enc_src_ip 10.12.0.8 enc_key_id 60987 enc_dst_port 4789 enc_tos 0 ip_flags nofrag in_hw in_hw_count 2 action order 1: tunnel_key unset pipe index 2 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth15) stolen index 2 ref 1 bind 1 cookie 5d44cdf6754fc237cc0bd688a6fe70a8 no_percpu used_hw_stats delayed ``` Am I doing something wrong? Is there anyone who has other results? -- <https://www.mirantis.com/> Vasyl Saienko Principal DevOps Engineer vsaienko(a)mirantis.com <dstoltenberg(a)mirantis.com> +(380) 66 072 07 17 <++1+(650)+564+7038>

3 3

Openstack integration with openid as leycloak
by amine belroul 30 Jul '25

30 Jul '25

Hello, Is there any detailed docs can help me to integrate openstack with openid? Please multi domains and multi projects.

3 2

[Cinder] pbm configuring multipath on NVMeoF
by Vincent Godin 30 Jul '25

30 Jul '25

Hello guys, Openstack: 2025.1 OS: Ubuntu 24.04 We are having trouble configuring multipath on a NetApp backend using NVMe over TCP. After reading numerous articles on this issue, we concluded that it was necessary to operate in native multipath mode, which is enabled by default on Ubuntu 24.04, and that it is no longer necessary to keep the multipathd service active for this to work. We ran inconclusive tests by setting the "use_multipath_for_image_xfer" and "enforce_multipath_for_image_xfer" variables to true in cinder.conf. We also set the "volume_use_multipath variable" to true in the libvirt section of nova.conf, but without success. After creating an instance and a volume on a server with Openstack, when querying the subsystem with the nvme command, we only get a single path. But when we query the backend from the same server with "nvme discover," we do get four paths. In the os-brick code, we see that it checks a Multipath variable that must be set to true... I think this is purely a configuration issue, but the examples given by various manufacturers (NetApp, PureStorage, etc.) don't indicate anything in particular. So, does anyone know the configuration to apply for multipath to work (cinder/nova)? Thank you.

3 8

OpenStack for AI Whitepaper
by Kendall Nelson 29 Jul '25

29 Jul '25

Hello Folks! As you might have heard, the OpenInfra AI WG has been starting to put together an OpenStack for AI Whitepaper and we wanted to formally invite you all to attend the WG's meetings and working sessions, to get involved on working group's the mailing list <https://lists.openinfra.org/mailman3/lists/ai-openstack-wg.lists.openinfra.…> and to help us shape the whitepaper into a resource for those looking to understand just how OpenStack is capable of serving AI workloads. We could really benefit from your innate understanding of how the OpenStack services work and also hope to surface feedback and questions from the group for the larger OpenStack community in future meetings. Also, we have organizations volunteering to do show and tell sessions about their OpenStack clouds hosting AI workloads and discussing pain points, lessons learned etc. The next of those show and tell sessions will take place on September 29. We have a meeting coming this week on Thursday at 13 UTC in this google meet <http://meet.google.com/wzn-jgcn-cfj>. If you would like an actual calendar invite, please reach out and I will make sure you are added! -- Kendall Nelson *Senior Upstream Developer Advocate* The OpenInfra Foundation <https://openinfra.dev/>

1 1

[kolla] IRC meeting cancelled - 30-th of July 2025
by Bartosz Bezak 29 Jul '25

29 Jul '25

Hello, Due to limited availability of the Kolla core team members, tomorrow’s Kolla IRC meeting is cancelled. Best regards, Bartosz Bezak

1 0

[publiccloud-sig] Public Cloud SIG meeting d.d. 30 July
by Kees Meijs | Nefos 28 Jul '25

28 Jul '25

Hello Stackers, Tomorrow (that's 30 July) the bi-weekly Public Cloud SIG meeting will be held again. The publicly available video meeting will be conducted from 07:00 hrs UTC (09:00 hrs CEST) and on here: https://meetpad.opendev.org/publiccloud-sig Additional information to be found on https://wiki.openstack.org/wiki/PublicCloudSIG See you tomorrow! Cheers, Kees __ Kees Meijs BICT Nefos Cloud & IT <https://nefos.com/contact> Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931 +31 (0)88 2088 188 <tel:+31882088188> nefos.com <https://nefos.com/contact> The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended. General terms and conditions ("The NLdigital Terms") apply to all our products and services.

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.2/R-9)
by Goutham Pacha Ravi 28 Jul '25

28 Jul '25

Hello Stackers, We're a month away from the feature freeze date for the 2025.2 ("Flamingo") release cycle [1], scheduled for 2025-08-28. OpenStack project teams are observing various internal deadlines prior to that important date to ensure a complete and timely release. As we look ahead, OpenStack's election officials will begin accepting self-nominations for the upcoming Project Team Lead roles and four seats on the OpenStack Technical Committee starting 2025-08-06 [2]. The nomination deadline is two weeks later, on 2025-08-20. You are welcome to submit an early nomination [3] now if you anticipate being away during the nomination window. To be a Project Team Lead candidate, you must be an Active Project Contributor—i.e., someone who has contributed to the project you wish to lead in the past 12 months (counted up to 2025-08-20). Any individual member of the OpenInfra Foundation, regardless of contributor status, may propose their candidacy for the OpenStack Technical Committee. To vote in these elections, you must be a member of the OpenInfra Foundation [4] and an active contributor—either through merged contributions to official projects or via nomination by a project’s PTL, liaisons, any Special Interest Group, or the OpenStack Technical Committee [5]. To receive your ballot, please ensure your email address is added in Gerrit and that you've opted in to receive emails from CIVS before 2025-08-20: https://civs1.civs.us/cgi-bin/opt_in.pl In the past week, the TC approved revised milestones for the cross-project goal of removing the use of Eventlet [6]. We still aim to drop Eventlet usage completely in the 2027.2 ("J") release. The "aetos" project now has a canonical service type called "metric-storage" [7]. Several other governance changes are currently under community review. === Weekly Meeting === The last weekly meeting of the OpenStack TC took place [8] on 2025-07-22, with a smaller attendance than usual. We expect this trend to continue through the summer break in the northern hemisphere. Key updates included changes to the Eventlet-removal goal timeline and a deeper look into the prolonged inactivity of the Monasca project team. While volunteers have consistently expressed interest in maintaining the project, it hasn’t met reactivation milestones since its last active cycle (2024.1). However, some improvements have been merged across the team’s repositories. The TC agreed to enforce a more formal and transparent process: projects must explicitly request extensions to inactive status [9], and future documentation will include retirement deadlines that compel TC votes on any extensions. We'll continue discussing the next steps for Monasca through governance proposals and this mailing list. Opendev administrators informed us that they completed the transition off Nodepool, and Debian Trixie images are nearly ready for testing. This is also useful for transitioning the Ceph CI job to Debian to enable faster fixes in the client software. In open discussion, the group revisited the definition of corporate affiliation in the TC charter. A proposal will be made to clarify what affiliation diversity means and how/why it should be disclosed during elections. The next meeting of the OpenStack Technical Committee is today, 2025-07-29. It will be hosted on the #openstack-tc channel on OFTC at 1700 UTC. Please find the meeting's agenda and other details on its wiki page [10]. I hope you'll be able to join us. === Governance Proposals === === Merged === - Make Eventlet removal deadlines more acceptable for operators | https://review.opendev.org/c/openstack/governance/+/952903 - Retire networking-midonet | https://review.opendev.org/c/openstack/governance/+/955364 === Open for review === - Define "affiliation" within the context of the TC | https://review.opendev.org/c/openstack/governance/+/956024 - Add series names to runtimes doc | https://review.opendev.org/c/openstack/governance/+/955810 - Remove Monasca from active project | https://review.opendev.org/c/openstack/governance/+/953671 - os-test-images never releases | https://review.opendev.org/c/openstack/governance/+/954249 - Require declaration of affiliation from TC Candidates | https://review.opendev.org/c/openstack/governance/+/949432 === Upcoming Events === - 2025-08-06: Nominations open for OpenStack elections: https://governance.openstack.org/election/ - 2025-08-20: Nominations close for OpenStack elections - 2025-08-26: OpenInfra Days, Korea: https://2025.openinfradays.kr/ - 2025-08-28: Feature Freeze deadline, Milestone-3 of the Flamingo release [1] - 2025-08-29: Colombia OpenInfra User Group Meetup: https://www.meetup.com/colombia-openinfra-user-group/events/307096751/ - 2025-10-17: OpenInfra Summit, Paris-Saclay: https://summit2025.openinfra.org/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.2 "Flamingo" Release Schedule: https://releases.openstack.org/flamingo/schedule.html [2] 2026.1 OpenStack Elections: https://governance.openstack.org/election/ [3] Submitting your candidacy: https://governance.openstack.org/election/#how-to-submit-a-candidacy [4] Join the OpenInfra Foundation: https://openinfra.org/join/individual/ [5] Who is an Active Contributor: https://governance.openstack.org/tc/reference/charter.html#voters-for-tc-se… [6] Eventlet removal timeline: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#com… [7] OpenStack service types: https://specs.openstack.org/openstack/service-types-authority/#service-data [8] TC Meeting IRC Log 2025-07-22: https://meetings.opendev.org/meetings/tc/2025/tc.2025-07-22-17.00.html [9] Emerging and inactive projects: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [10] TC Meeting Agenda, 2025-07-29: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[manila] Flamingo Mid Cycle: Aug 6 - 7th
by Carlos Silva 28 Jul '25

28 Jul '25

Hello, Zorillas and interested stackers, The OpenStack Manila's mid cycle event [1] will take place on Aug 6th and 7th, from 14 to 16 UTC. More details on the .ics file [1]. At the event, we will focus on checking on the progress of the deliverables we planned for this cycle, as well as do collaborative review sessions for the features being proposed to speed up the review process. If you have a topic you would like to discuss, please add them to the mid cycle etherpad [2]. [1] https://gist.github.com/silvacarloss/598ad6d311cfe0a588b6b47ea979dff5 [2] https://etherpad.opendev.org/p/manila-flamingo-mid-cycle Looking forward to it, carloss

1 0

[i18n] Sharing CMU Students’ Contributions: Progress on Zanata-to-Weblate Migration
by Ian Y. Choi 28 Jul '25

28 Jul '25

Dear OpenStack Community, Over the past six weeks, Seongsoo and I had the privilege of mentoring a group of four students from Carnegie Mellon University (CMU) [1] as part of our ongoing effort to migrate the OpenStack Internationalization (I18n) platform from Zanata to Weblate. I would like to take this opportunity to highlight the significant and meaningful contributions made by three students, Furqan Saeed, Simon Yu, and Jana Hussien, who recently completed video demos showcasing their work [2] [3]. Project Context As many may know, the I18n SIG has been working to modernize our translation infrastructure by transitioning from Zanata (now unmaintained) to Weblate — an actively developed, open source platform. This project not only ensures the sustainability of our translations, but also offers a valuable learning path for new contributors. Below are the key outcomes achieved by Furqan, Simon and Jana: 1. i18n tools script test codes for Weblate (by Furqan Saeed & Simon Yu) Furqan and Simon focused on bringing robust, production-quality unit testing to the collection of Python scripts and tools responsible for Weblate integration. Furqan’s video [2] demonstrates these efforts. Together, they: Developed a comprehensive suite of ~34 unit tests covering the main Weblate tool files. Used Python’s unittest and mocking strategies to ensure functions behave as expected, even simulating server responses. Noted and addressed gaps in requirements, such as missing library dependencies. Provided clear documentation (README.rst) explaining test infrastructure and file structure. Gained and demonstrated familiarity with OpenStack’s development guidelines, Gerrit workflows, and CI requirements. Their efforts establish a much needed test baseline for future improvements and bring our I18n tooling closer to OpenStack’s best practices. This contribution not only enhances software reliability but also creates a safer, more efficient environment for future contributors. 2. Migration of Zuul infrastructure scripts to Weblate (by Jana Hussien) Jana concentrated on migrating and refactoring the automation scripts that enable daily translation updates in Zuul, shifting from Zanata to Weblate. Her work involved: Analyzing and translating the logic in current scripts to ensure seamless migration, including first-time migration routines for new projects. Developing new scripts to automate creation of Weblate components and upload of translation templates (POT) and PO files; handling language code transformations between platforms. Troubleshooting integration challenges between Weblate’s API/CLI and OpenStack workflows, identifying and overcoming tricky configuration issues related to version control structure. Successfully demonstrating the migration workflow for projects such as trove-dashboard and horizon, with detailed documentation of results. Improving operational resilience by implementing enhanced error handling and summary reporting in the scripts. Discovering and addressing several issues during live testing, and documenting workarounds and plans for future improvements. Jana has also outlined next steps, including integrating these scripts further into the Zuul pipeline, modularizing code, and enhancing public documentation. These contributions represent valuable progress toward a full, sustainable migration to Weblate - benefiting the entire OpenStack community by advancing our translation workflows and broadening global accessibility. On behalf of all mentors (myself, Seongsoo Cho, Goutham Pacha Ravi), I’d like to thank Furqan, Simon, and Jana for their hard work, dedication, and collaborative spirit, and encourage other students and new contributors to get involved. If anyone would like to review their code, documentation, or get involved in this effort, please feel free to reach out or interact directly with their contributions [6-9]! Best regards, Ian Y. Choi (ianychoi) & Seongsoo Cho (seongsoocho) OpenStack I18n SIG & Student Mentors [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] Video - OpenStack I18n Zanata-to-Weblate Migration: Weblate Tools Unit Test Development (CMU Project Demo) (Furqan), link: https://youtu.be/w966IpBCqeE [3] Video - OpenStack I18n Zanata-to-Weblate Migration: Translation Scripts working on Zuul (CMU Project Demo) (Jana), link: https://youtu.be/RcQweOe-vfo [4] Photo of our kick off meeting: https://photos.app.goo.gl/QhNwBusFZjwvTBzi7 [5] Photos of our final mentoring meeting: Final Mentoring Meeting - https://1drv.ms/f/c/6cc85a64cb418204/EljY3AfrnypGrF6EyDZq83MBDZpLTPt-8VPymj… [6] https://review.opendev.org/c/openstack/i18n/+/953745 (by Furqan, review is welcome) [7] https://review.opendev.org/c/openstack/i18n/+/955982 (by Simon, review is welcome) [8] https://review.opendev.org/c/openstack/i18n/+/955439 (by Jana, working in progress) [9] https://review.opendev.org/c/openstack/trove-dashboard/+/952787 (by Jana, merged)

1 0