- openstack-discuss - lists.openstack.org

[openstack][magnum]Cannot upgrade clusters
by Nguyễn Hữu Khôi 31 Jul '25

31 Jul '25

Hello. I cannot upgrade clusters via magnum_cluster_api. These are log from magnum_api: https://anotepad.com/notes/c9h8bd55 OPS: 2025.1 OS: Ubuntu 24.04 magnum_cluster_api: v0.30.0 Would you give some ideas on it? Thank you! Nguyen Huu Khoi

1 1

[Neutron][VPNaaS] Multiple tunnels per IPSec Service
by Shubham Yadav 31 Jul '25

31 Jul '25

Hello OpenStack Community, I'm working with OpenStack VPNaaS (using strongSwan as the backend driver) and have a question about tunnel configuration capabilities. *Current Setup:* OpenStack VPNaaS with strongSwan driver (StrongSwanDriver) Single VPN Service (gateway) with multiple IPSec Site Connections Each connection targets different remote peers with distinct subnets *Configuration Details:* VPN Service: Connected to a single router Multiple IPSec Site Connections under the same VPN Service: Connection 1: To MY office (10.10.100.0/24) via peer 68.x.y.34 Connection 2: To Rackspace (172.24.60.0/22) via peer 184.n.p.53 Shared IKE and IPSec policies across connections Local endpoint groups covering multiple subnets (192.168.0.0/24, 10.43.0.0/16) *Question:*Is it supported/recommended to have multiple IPSec Site Connections under a single VPN Service? I understand this creates multiple tunnels from the same OpenStack router to different remote peers. *Current Issue:*One of my connections shows "PENDING_CREATE" status while the other is "ACTIVE". I'm trying to determine if this is: A configuration issue on my end A limitation of having multiple tunnels per VPN service A strongSwan-specific behavior in the OpenStack context *Specific Questions:* Does VPNaaS officially support multiple IPSec connections per VPN Service? Are there any known limitations or best practices when using multiple tunnels? Should each remote site have its own dedicated VPN Service instead? I've checked the documentation but couldn't find clear guidance on multi-tunnel scenarios. Any insights from the community would be greatly appreciated. *Environment:* OpenStack version: 2024.2 Dalmatian Neutron VPNaaS with strongSwan backend Kubernetes-deployed OpenStack (OpenStack-Helm) Thank you for your time and assistance! Best regards, Shubham

1 0

[Neutron][VPNaaS] Multiple tunnels per IPSec Service
by Shubham Yadav 31 Jul '25

31 Jul '25

Hello OpenStack Community, I'm working with OpenStack VPNaaS (using strongSwan as the backend driver) and have a question about tunnel configuration capabilities. Current Setup: OpenStack VPNaaS with strongSwan driver (StrongSwanDriver) Single VPN Service (gateway) with multiple IPSec Site Connections Each connection targets different remote peers with distinct subnets Configuration Details: VPN Service: Connected to a single router Multiple IPSec Site Connections under the same VPN Service: Connection 1: To MY office (10.10.100.0/24) via peer 68.x.y.34 Connection 2: To Rackspace (172.24.60.0/22) via peer 184.n.p.53 Shared IKE and IPSec policies across connections Local endpoint groups covering multiple subnets (192.168.0.0/24, 10.43.0.0/16) Question: Is it supported/recommended to have multiple IPSec Site Connections under a single VPN Service? I understand this creates multiple tunnels from the same OpenStack router to different remote peers. Current Issue: One of my connections shows "PENDING_CREATE" status while the other is "ACTIVE". I'm trying to determine if this is: A configuration issue on my end A limitation of having multiple tunnels per VPN service A strongSwan-specific behavior in the OpenStack context Specific Questions: Does VPNaaS officially support multiple IPSec connections per VPN Service? Are there any known limitations or best practices when using multiple tunnels? Should each remote site have its own dedicated VPN Service instead? I've checked the documentation but couldn't find clear guidance on multi-tunnel scenarios. Any insights from the community would be greatly appreciated. Environment: OpenStack version: 2024.2 Dalmatian Neutron VPNaaS with strongSwan backend Kubernetes-deployed OpenStack (OpenStack-Helm) Thank you for your time and assistance! Best regards, Shubham

1 0

[Ovs-dpdk]Issue with ovs-dpdk deployment in Kolla-Ansible (zed)
by samira_darvishnia＠yahoo.com 31 Jul '25

31 Jul '25

Hi all, I'm trying to deploy OVS-DPDK using Kolla-Ansible Zed, and I'm running into a blocking issue during deployment. I would really appreciate your insights or suggestions. Problem Summary: During the deployment of ovs-dpdk, the container ovsdpdk-vswitchd fails to start, with the following error in its logs: + echo 'Running command: '\''/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile'\''' + exec /usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile Running command: '/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile' ovs-vswitchd: invalid option -- 'w' ovs-vswitchd: invalid option -- 'w' EAL: FATAL: Invalid 'command line' arguments. 2025-07-30T13:44:59Z|00017|dpdk|EMER|Unable to initialize DPDK: Invalid argument ovs-vswitchd: Cannot init EAL (Invalid argument) This appears to be related to an incorrect parsing of arguments for ovs-vswitchd. What I’ve tried: I already applied this patch (as referenced in Gerrit): https://review.opendev.org/c/openstack/kolla-ansible/+/913653/1/ansible/rol… However, the issue still persists. Additional context: • Deployment hangs on the task: name: Wait for dpdk tunnel ip It waits for 5 minutes and then fails. • The parameter dpdk_tunnel_interface_address is already set in my global.yml. • I’ve configured and bound NICs to vfio-pci. • Environment info: • OpenStack Zed • Kolla-Ansible Zed • dpdk_tunnel_interface_address is set • OVS-DPDK is containerized via Kolla Has anyone seen this issue before or can suggest further steps? Any insight on how to correctly pass EAL arguments or what else might be misconfigured would be very helpful. Thanks in advance, Samira

1 0

[Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches �C Suspected Regression in Neutron Master Branch
by Renu Rani 31 Jul '25

31 Jul '25

Hi Neutron Team, We are currently facing an issue in Tacker Zuul patch jobs build (with master branch), where the DevStack setup fails during the initial Neutron network creation step. Problem statement: The failure occurs at this point in the stack.sh script, for devstack environment creation in build process. Below command execution is failing in stack.sh [openstack --os-cloud devstack-admin-demo --os-region RegionOne router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc d777d04b-93ff-4e94-b6a9-cc8de3e23921] Error Message: This results in a 500 Internal Server Error, as seen in the logs: [HttpException: 500: Server Error for url: http://10.x.x.x/networking/v2.0/routers/20b264df-5f19-4599-a83d-2e76e7cbf5d…] Analysis for the issue: Below error is coming in Neutron API logs: [ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on un-acquired lock] 1. Tried overriding the required-projects in Zuul to use stable/2025.1 branches (instead of master) to identify project(master branch) which is causing root cause. 2. Tacker team has created DNM patch (952892) to invetsigate this in detail. https://review.opendev.org/c/openstack/tacker/+/952892 1. In DNM patch Patchset 11 (on June 29), we have taken devstack stable/2025.1. In stable branch the devstack environment created sucessfully. 2. In DNM patch Patchset 25 (on July 8), with the same devstack override (stable/2025.1), the job now fails earlier during DevStack setup. The logs show that Neutron failed to start, causing the job to fail across all nodes. As per observations and anlysis seems the recent changes in the Neutron master branch between June 29 and July 8 might be causing this issue. We’d appreciate your help in reviewing the changes in Neutron's master branch during that window that could cause this behavior and suggesting any debugging approaches or patches we can test on tacker. Please let us know if you'd like additional logs or info from the Zuul job runs. Thanks in advance for your support! Thanks & Regards, Renu Rani (レヌラニ) Mobile: +91-9311888962 E-mail: renu.rani(a)india.nec.com<mailto:renu.rani@india.nec.com> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NEC Corporation India Private Limited or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NEC Corporation India Private Limited or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.

5 10

[neutron] Is NVIDIA ASAP2 offloading with OpenStack just a myph?
by Vasyl Saienko 31 Jul '25

31 Jul '25

Hello OpenStack community, I'm trying to set up OpenStack with Nvidia ASAP2 hardware offloading in switchdev mode according to OpenStack guidelines <https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html> and Nvidia guidelines <https://docs.nvidia.com/doca/sdk/ovs-kernel+hardware+offloads/index.html> on the ConnectX6DX. I've tried different network types VLAN and VXLAN but offloading for none of them works. From an OpenStack standpoint everything is configured according to documentation, representor device is created and plugged in the br-int bridge (eth14) physical. PF enp33s0f0np0 is set to switchdev mode ``` devlink dev eswitch show pci/0000:21:00.0 pci/0000:21:00.0: mode switchdev inline-mode none encap-mode basic ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl show 12e00cdb-6520-4666-908f-57f59e8b9bd4 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-ex Interface br-ex type: internal Port pr-floating Interface pr-floating Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port int-br-tenant Interface int-br-tenant type: patch options: {peer=phy-br-tenant} Port tap58ac2b17-fb tag: 1 Interface tap58ac2b17-fb Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port eth14 tag: 1 Interface eth14 Bridge br-tenant Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tenant Interface br-tenant type: internal Port phy-br-tenant Interface phy-br-tenant type: patch options: {peer=int-br-tenant} Port enp33s0f0np0 Interface enp33s0f0np0 Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} ovs_version: "3.0.0-0056-25.04-based-3.3.5" ``` But hardware offloading is not happening, tcpdump on eth14 still shows all packets, and no TC rules are created. OpenStack uses complex openvswitch rules to setup virtual networking, in this scenario vlan tagging is done by the following rule ``` cookie=0x45b91976d5c4f175, duration=233.443s, table=60, n_packets=314, n_bytes=35574, priority=100,in_port=eth14 actions=set_field:0x11->reg5,set_field:0x1->reg6,resubmit(,73) ``` If we simplify openflow rules and align them with Nvidia guidelines for openvswitch hardware offloading by plugging representor device directly into physnet bridge, and assign vlan tag as port attribute hardware offloading starts working. ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl del-port eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl add-port br-tenant eth14 root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# ovs-vsctl set port eth14 tag=1443 ``` ``` root@kaas-node-52648a8f-cbfd-4b9a-b392-1594b1d97cfa:~# for i in $(ip link | awk '{print $2}' |grep -e enp -e ens |grep -v vf$); do tc filter show dev $i ingress; done filter protocol 802.1Q pref 6 flower chain 0 filter protocol 802.1Q pref 6 flower chain 0 handle 0x1 vlan_id 1443 vlan_prio 0 vlan_ethtype arp dst_mac fa:16:3e:36:1b:90 src_mac fa:16:3e:39:90:d8 eth_type arp in_hw in_hw_count 1 action order 1: vlan pop pipe index 1 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth14) stolen index 1 ref 1 bind 1 cookie 4b5b5a701f4e3fee7ac1c198037522e3 no_percpu used_hw_stats delayed ``` Similar issue occurs when using the VXLAN network. OpenStack setup VXLAN tunnel like, where tunnel_id is inserted dynamically via openflow rules (in_key=flow) ``` Port vxlan-0a0c0004 Interface vxlan-0a0c0004 type: vxlan options: {df_default="true", dst_port="4790", egress_pkt_mark="0", in_key=flow, local_ip="10.12.0.9", out_key=flow, remote_ip="10.12.0.4"} ``` In this configuration hardware offloading is not happening neither. But when setting up vxlan according to nvidia guidelines (no complex rules are involved) hardware offloading starts working (tcpdump for representor device shows only first packet and tc rules are being created). ``` Port vxlan3 Interface vxlan3 type: vxlan options: {key="60987", local_ip="10.12.0.9", remote_ip="10.12.0.4"} ``` ``` Cannot find device "enp33s0f0npf0vf" Cannot find device "veth68d427d@if1" filter protocol ip pref 2 flower chain 0 filter protocol ip pref 2 flower chain 0 handle 0x1 dst_mac fa:16:3e:65:0c:6d src_mac 52:5a:d0:9e:8a:32 eth_type ipv4 enc_dst_ip 10.12.0.9 enc_src_ip 10.12.0.8 enc_key_id 60987 enc_dst_port 4789 enc_tos 0 ip_flags nofrag in_hw in_hw_count 2 action order 1: tunnel_key unset pipe index 2 ref 1 bind 1 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device eth15) stolen index 2 ref 1 bind 1 cookie 5d44cdf6754fc237cc0bd688a6fe70a8 no_percpu used_hw_stats delayed ``` Am I doing something wrong? Is there anyone who has other results? -- <https://www.mirantis.com/> Vasyl Saienko Principal DevOps Engineer vsaienko(a)mirantis.com <dstoltenberg(a)mirantis.com> +(380) 66 072 07 17 <++1+(650)+564+7038>

3 3

Openstack integration with openid as leycloak
by amine belroul 30 Jul '25

30 Jul '25

Hello, Is there any detailed docs can help me to integrate openstack with openid? Please multi domains and multi projects.

3 2

[Cinder] pbm configuring multipath on NVMeoF
by Vincent Godin 30 Jul '25

30 Jul '25

Hello guys, Openstack: 2025.1 OS: Ubuntu 24.04 We are having trouble configuring multipath on a NetApp backend using NVMe over TCP. After reading numerous articles on this issue, we concluded that it was necessary to operate in native multipath mode, which is enabled by default on Ubuntu 24.04, and that it is no longer necessary to keep the multipathd service active for this to work. We ran inconclusive tests by setting the "use_multipath_for_image_xfer" and "enforce_multipath_for_image_xfer" variables to true in cinder.conf. We also set the "volume_use_multipath variable" to true in the libvirt section of nova.conf, but without success. After creating an instance and a volume on a server with Openstack, when querying the subsystem with the nvme command, we only get a single path. But when we query the backend from the same server with "nvme discover," we do get four paths. In the os-brick code, we see that it checks a Multipath variable that must be set to true... I think this is purely a configuration issue, but the examples given by various manufacturers (NetApp, PureStorage, etc.) don't indicate anything in particular. So, does anyone know the configuration to apply for multipath to work (cinder/nova)? Thank you.

3 8

OpenStack for AI Whitepaper
by Kendall Nelson 30 Jul '25

30 Jul '25

Hello Folks! As you might have heard, the OpenInfra AI WG has been starting to put together an OpenStack for AI Whitepaper and we wanted to formally invite you all to attend the WG's meetings and working sessions, to get involved on working group's the mailing list <https://lists.openinfra.org/mailman3/lists/ai-openstack-wg.lists.openinfra.…> and to help us shape the whitepaper into a resource for those looking to understand just how OpenStack is capable of serving AI workloads. We could really benefit from your innate understanding of how the OpenStack services work and also hope to surface feedback and questions from the group for the larger OpenStack community in future meetings. Also, we have organizations volunteering to do show and tell sessions about their OpenStack clouds hosting AI workloads and discussing pain points, lessons learned etc. The next of those show and tell sessions will take place on September 29. We have a meeting coming this week on Thursday at 13 UTC in this google meet <http://meet.google.com/wzn-jgcn-cfj>. If you would like an actual calendar invite, please reach out and I will make sure you are added! -- Kendall Nelson *Senior Upstream Developer Advocate* The OpenInfra Foundation <https://openinfra.dev/>

1 1

[kolla] IRC meeting cancelled - 30-th of July 2025
by Bartosz Bezak 29 Jul '25

29 Jul '25

Hello, Due to limited availability of the Kolla core team members, tomorrow’s Kolla IRC meeting is cancelled. Best regards, Bartosz Bezak

1 0