- openstack-discuss - lists.openstack.org

[Ovs-dpdk]Issue with ovs-dpdk deployment in Kolla-Ansible (zed)
by samira_darvishnia＠yahoo.com 31 Jul '25

31 Jul '25

Hi all, I'm trying to deploy OVS-DPDK using Kolla-Ansible Zed, and I'm running into a blocking issue during deployment. I would really appreciate your insights or suggestions. Problem Summary: During the deployment of ovs-dpdk, the container ovsdpdk-vswitchd fails to start, with the following error in its logs: + echo 'Running command: '\''/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile'\''' + exec /usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile Running command: '/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile' ovs-vswitchd: invalid option -- 'w' ovs-vswitchd: invalid option -- 'w' EAL: FATAL: Invalid 'command line' arguments. 2025-07-30T13:44:59Z|00017|dpdk|EMER|Unable to initialize DPDK: Invalid argument ovs-vswitchd: Cannot init EAL (Invalid argument) This appears to be related to an incorrect parsing of arguments for ovs-vswitchd. What I’ve tried: I already applied this patch (as referenced in Gerrit): https://review.opendev.org/c/openstack/kolla-ansible/+/913653/1/ansible/rol… However, the issue still persists. Additional context: • Deployment hangs on the task: name: Wait for dpdk tunnel ip It waits for 5 minutes and then fails. • The parameter dpdk_tunnel_interface_address is already set in my global.yml. • I’ve configured and bound NICs to vfio-pci. • Environment info: • OpenStack Zed • Kolla-Ansible Zed • dpdk_tunnel_interface_address is set • OVS-DPDK is containerized via Kolla Has anyone seen this issue before or can suggest further steps? Any insight on how to correctly pass EAL arguments or what else might be misconfigured would be very helpful. Thanks in advance, Samira

1 0

[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
by Renu Rani 31 Jul '25

31 Jul '25

Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5d…] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌラニ) Mobile: +91-9311888962 E-mail: renu.rani(a)india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.

5 10

[neutron] Is NVIDIA ASAP2 offloading with OpenStack just a myph?
by Vasyl Saienko 30 Jul '25

30 Jul '25

Hello OpenStack community, I'm trying to set up OpenStack with Nvidia ASAP2 hardware offloading in switchdev mode according to OpenStack guidelines <https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html> and Nvidia guidelines <https://docs.nvidia.com/doca/sdk/ovs-kernel+hardware+offloads/index.html> on the ConnectX6DX. I've tried different network types VLAN and VXLAN but offloading for none of them works. From an OpenStack standpoint everything is configured according to documentation, representor device is created and plugged in the br-int bridge (eth14) physical. PF enp33s0f0np0 is set to switchdev mode ``` devlink dev eswitch show pci/0000:21:00.0 pci/0000:21:00.0: mode switchdev inline-mode none encap-mode basic ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl show 12e00cdb-6520-4666-908f-57f59e8b9bd4 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-ex Interface br-ex type: internal Port pr-floating Interface pr-floating Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port int-br-tenant Interface int-br-tenant type: patch options: {peer=phy-br-tenant} Port tap58ac2b17-fb tag: 1 Interface tap58ac2b17-fb Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port eth14 tag: 1 Interface eth14 Bridge br-tenant Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tenant Interface br-tenant type: internal Port phy-br-tenant Interface phy-br-tenant type: patch options: {peer=int-br-tenant} Port enp33s0f0np0 Interface enp33s0f0np0 Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} ovs_version: "3.0.0-0056-25.04-based-3.3.5" ``` But hardware offloading is not happening, tcpdump on eth14 still shows all packets, and no TC rules are created. OpenStack uses complex openvswitch rules to setup virtual networking, in this scenario vlan tagging is done by the following rule ``` cookie=0x45b91976d5c4f175, duration=233.443s, table=60, n_packets=314, n_bytes=35574, priority=100,in_port=eth14 actions=set_field:0x11->reg5,set_field:0x1->reg6,resubmit(,73) ``` If we simplify openflow rules and align them with Nvidia guidelines for openvswitch hardware offloading by plugging representor device directly into physnet bridge, and assign vlan tag as port attribute hardware offloading starts working. ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl del-port eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl add-port br-tenant eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl set port eth14 tag=1443 ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# for i in $(ip link | awk '{print $2}' |grep -e enp -e ens |grep -v vf$); do tc filter show dev $i ingress; done filter protocol 802.1Q pref 6 flower chain 0 filter protocol 802.1Q pref 6 flower chain 0 handle 0x1 vlan_id 1443 vlan_prio 0 vlan_ethtype arp dst_mac fa:16:3e:36:1b:90 src_mac fa:16:3e:39:90:d8 eth_type arp in_hw in_hw_count 1 action order 1: vlan pop pipe index 1 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth14) stolen index 1 ref 1 bind 1 cookie 4b5b5a701f4e3fee7ac1c198037522e3 no_percpu used_hw_stats delayed ``` Similar issue occurs when using the VXLAN network. OpenStack setup VXLAN tunnel like, where tunnel_id is inserted dynamically via openflow rules (in_key=flow) ``` Port vxlan-0a0c0004 Interface vxlan-0a0c0004 type: vxlan options: {df_default="true", dst_port="4790", egress_pkt_mark="0", in_key=flow, local_ip="10.12.0.9", out_key=flow, remote_ip="10.12.0.4"} ``` In this configuration hardware offloading is not happening neither. But when setting up vxlan according to nvidia guidelines (no complex rules are involved) hardware offloading starts working (tcpdump for representor device shows only first packet and tc rules are being created). ``` Port vxlan3 Interface vxlan3 type: vxlan options: {key="60987", local_ip="10.12.0.9", remote_ip="10.12.0.4"} ``` ``` Cannot find device "enp33s0f0npf0vf" Cannot find device "veth68d427d@if1" filter protocol ip pref 2 flower chain 0 filter protocol ip pref 2 flower chain 0 handle 0x1 dst_mac fa:16:3e:65:0c:6d src_mac 52:5a:d0:9e:8a:32 eth_type ipv4 enc_dst_ip 10.12.0.9 enc_src_ip 10.12.0.8 enc_key_id 60987 enc_dst_port 4789 enc_tos 0 ip_flags nofrag in_hw in_hw_count 2 action order 1: tunnel_key unset pipe index 2 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth15) stolen index 2 ref 1 bind 1 cookie 5d44cdf6754fc237cc0bd688a6fe70a8 no_percpu used_hw_stats delayed ``` Am I doing something wrong? Is there anyone who has other results? -- <https://www.mirantis.com/> Vasyl Saienko Principal DevOps Engineer vsaienko(a)mirantis.com <dstoltenberg(a)mirantis.com> +(380) 66 072 07 17 <++1+(650)+564+7038>

3 3

Openstack integration with openid as leycloak
by amine belroul 30 Jul '25

30 Jul '25

Hello, Is there any detailed docs can help me to integrate openstack with openid? Please multi domains and multi projects.

3 2

[Cinder] pbm configuring multipath on NVMeoF
by Vincent Godin 30 Jul '25

30 Jul '25

Hello guys, Openstack: 2025.1 OS: Ubuntu 24.04 We are having trouble configuring multipath on a NetApp backend using NVMe over TCP. After reading numerous articles on this issue, we concluded that it was necessary to operate in native multipath mode, which is enabled by default on Ubuntu 24.04, and that it is no longer necessary to keep the multipathd service active for this to work. We ran inconclusive tests by setting the "use_multipath_for_image_xfer" and "enforce_multipath_for_image_xfer" variables to true in cinder.conf. We also set the "volume_use_multipath variable" to true in the libvirt section of nova.conf, but without success. After creating an instance and a volume on a server with Openstack, when querying the subsystem with the nvme command, we only get a single path. But when we query the backend from the same server with "nvme discover," we do get four paths. In the os-brick code, we see that it checks a Multipath variable that must be set to true... I think this is purely a configuration issue, but the examples given by various manufacturers (NetApp, PureStorage, etc.) don't indicate anything in particular. So, does anyone know the configuration to apply for multipath to work (cinder/nova)? Thank you.

3 8

OpenStack for AI Whitepaper
by Kendall Nelson 30 Jul '25

30 Jul '25

Hello Folks! As you might have heard, the OpenInfra AI WG has been starting to put together an OpenStack for AI Whitepaper and we wanted to formally invite you all to attend the WG's meetings and working sessions, to get involved on working group's the mailing list <https://lists.openinfra.org/mailman3/lists/ai-openstack-wg.lists.openinfra.…> and to help us shape the whitepaper into a resource for those looking to understand just how OpenStack is capable of serving AI workloads. We could really benefit from your innate understanding of how the OpenStack services work and also hope to surface feedback and questions from the group for the larger OpenStack community in future meetings. Also, we have organizations volunteering to do show and tell sessions about their OpenStack clouds hosting AI workloads and discussing pain points, lessons learned etc. The next of those show and tell sessions will take place on September 29. We have a meeting coming this week on Thursday at 13 UTC in this google meet <http://meet.google.com/wzn-jgcn-cfj>. If you would like an actual calendar invite, please reach out and I will make sure you are added! -- Kendall Nelson *Senior Upstream Developer Advocate* The OpenInfra Foundation <https://openinfra.dev/>

1 1

[kolla] IRC meeting cancelled - 30-th of July 2025
by Bartosz Bezak 29 Jul '25

29 Jul '25

Hello, Due to limited availability of the Kolla core team members, tomorrow’s Kolla IRC meeting is cancelled. Best regards, Bartosz Bezak

1 0

[publiccloud-sig] Public Cloud SIG meeting d.d. 30 July
by Kees Meijs | Nefos 29 Jul '25

29 Jul '25

Hello Stackers, Tomorrow (that's 30 July) the bi-weekly Public Cloud SIG meeting will be held again. The publicly available video meeting will be conducted from 07:00 hrs UTC (09:00 hrs CEST) and on here: https://meetpad.opendev.org/publiccloud-sig Additional information to be found on https://wiki.openstack.org/wiki/PublicCloudSIG See you tomorrow! Cheers, Kees __ Kees Meijs BICT Nefos Cloud & IT <https://nefos.com/contact> Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931 +31 (0)88 2088 188 <tel:+31882088188> nefos.com <https://nefos.com/contact> The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended. General terms and conditions ("The NLdigital Terms") apply to all our products and services.

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.2/R-9)
by Goutham Pacha Ravi 29 Jul '25

29 Jul '25

Hello Stackers, We're a month away from the feature freeze date for the 2025.2 ("Flamingo") release cycle [1], scheduled for 2025-08-28. OpenStack project teams are observing various internal deadlines prior to that important date to ensure a complete and timely release. As we look ahead, OpenStack's election officials will begin accepting self-nominations for the upcoming Project Team Lead roles and four seats on the OpenStack Technical Committee starting 2025-08-06 [2]. The nomination deadline is two weeks later, on 2025-08-20. You are welcome to submit an early nomination [3] now if you anticipate being away during the nomination window. To be a Project Team Lead candidate, you must be an Active Project Contributor—i.e., someone who has contributed to the project you wish to lead in the past 12 months (counted up to 2025-08-20). Any individual member of the OpenInfra Foundation, regardless of contributor status, may propose their candidacy for the OpenStack Technical Committee. To vote in these elections, you must be a member of the OpenInfra Foundation [4] and an active contributor—either through merged contributions to official projects or via nomination by a project’s PTL, liaisons, any Special Interest Group, or the OpenStack Technical Committee [5]. To receive your ballot, please ensure your email address is added in Gerrit and that you've opted in to receive emails from CIVS before 2025-08-20: https://civs1.civs.us/cgi-bin/opt_in.pl In the past week, the TC approved revised milestones for the cross-project goal of removing the use of Eventlet [6]. We still aim to drop Eventlet usage completely in the 2027.2 ("J") release. The "aetos" project now has a canonical service type called "metric-storage" [7]. Several other governance changes are currently under community review. === Weekly Meeting === The last weekly meeting of the OpenStack TC took place [8] on 2025-07-22, with a smaller attendance than usual. We expect this trend to continue through the summer break in the northern hemisphere. Key updates included changes to the Eventlet-removal goal timeline and a deeper look into the prolonged inactivity of the Monasca project team. While volunteers have consistently expressed interest in maintaining the project, it hasn’t met reactivation milestones since its last active cycle (2024.1). However, some improvements have been merged across the team’s repositories. The TC agreed to enforce a more formal and transparent process: projects must explicitly request extensions to inactive status [9], and future documentation will include retirement deadlines that compel TC votes on any extensions. We'll continue discussing the next steps for Monasca through governance proposals and this mailing list. Opendev administrators informed us that they completed the transition off Nodepool, and Debian Trixie images are nearly ready for testing. This is also useful for transitioning the Ceph CI job to Debian to enable faster fixes in the client software. In open discussion, the group revisited the definition of corporate affiliation in the TC charter. A proposal will be made to clarify what affiliation diversity means and how/why it should be disclosed during elections. The next meeting of the OpenStack Technical Committee is today, 2025-07-29. It will be hosted on the #openstack-tc channel on OFTC at 1700 UTC. Please find the meeting's agenda and other details on its wiki page [10]. I hope you'll be able to join us. === Governance Proposals === === Merged === - Make Eventlet removal deadlines more acceptable for operators | https://review.opendev.org/c/openstack/governance/+/952903 - Retire networking-midonet | https://review.opendev.org/c/openstack/governance/+/955364 === Open for review === - Define "affiliation" within the context of the TC | https://review.opendev.org/c/openstack/governance/+/956024 - Add series names to runtimes doc | https://review.opendev.org/c/openstack/governance/+/955810 - Remove Monasca from active project | https://review.opendev.org/c/openstack/governance/+/953671 - os-test-images never releases | https://review.opendev.org/c/openstack/governance/+/954249 - Require declaration of affiliation from TC Candidates | https://review.opendev.org/c/openstack/governance/+/949432 === Upcoming Events === - 2025-08-06: Nominations open for OpenStack elections: https://governance.openstack.org/election/ - 2025-08-20: Nominations close for OpenStack elections - 2025-08-26: OpenInfra Days, Korea: https://2025.openinfradays.kr/ - 2025-08-28: Feature Freeze deadline, Milestone-3 of the Flamingo release [1] - 2025-08-29: Colombia OpenInfra User Group Meetup: https://www.meetup.com/colombia-openinfra-user-group/events/307096751/ - 2025-10-17: OpenInfra Summit, Paris-Saclay: https://summit2025.openinfra.org/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.2 "Flamingo" Release Schedule: https://releases.openstack.org/flamingo/schedule.html [2] 2026.1 OpenStack Elections: https://governance.openstack.org/election/ [3] Submitting your candidacy: https://governance.openstack.org/election/#how-to-submit-a-candidacy [4] Join the OpenInfra Foundation: https://openinfra.org/join/individual/ [5] Who is an Active Contributor: https://governance.openstack.org/tc/reference/charter.html#voters-for-tc-se… [6] Eventlet removal timeline: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#com… [7] OpenStack service types: https://specs.openstack.org/openstack/service-types-authority/#service-data [8] TC Meeting IRC Log 2025-07-22: https://meetings.opendev.org/meetings/tc/2025/tc.2025-07-22-17.00.html [9] Emerging and inactive projects: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [10] TC Meeting Agenda, 2025-07-29: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[manila] Flamingo Mid Cycle: Aug 6 - 7th
by Carlos Silva 28 Jul '25

28 Jul '25

Hello, Zorillas and interested stackers, The OpenStack Manila's mid cycle event [1] will take place on Aug 6th and 7th, from 14 to 16 UTC. More details on the .ics file [1]. At the event, we will focus on checking on the progress of the deliverables we planned for this cycle, as well as do collaborative review sessions for the features being proposed to speed up the review process. If you have a topic you would like to discuss, please add them to the mid cycle etherpad [2]. [1] https://gist.github.com/silvacarloss/598ad6d311cfe0a588b6b47ea979dff5 [2] https://etherpad.opendev.org/p/manila-flamingo-mid-cycle Looking forward to it, carloss

1 0