July 2025 - openstack-discuss - lists.openstack.org

[Zed][Neutron][OVN] Severe VM Network Performance Degradation Compared to Host
by Jeff Yang 05 Aug '25

05 Aug '25

Hi all, I'm running OpenStack Zed with OVN as the Neutron backend. I've encountered a severe VM network performance degradation issue and would appreciate any insights from the community. 🔹 **Environment** - OpenStack version: Zed - Neutron backend: OVN - Two VMs (each 4 vCPUs, 8 GB RAM) launched on two different compute nodes - Multi-queue is enabled on the VM vNICs 🔹 **Issue** The network performance inside the VMs is significantly worse than on the host nodes, with high packet retransmissions observed. 🔹 **Test results** **Host-to-Host (VTEP IPs)** ``` $ iperf3 -c 192.168.152.152 Connecting to host 192.168.152.152, port 5201 [ 5] local 192.168.152.153 port 45352 connected to 192.168.152.152 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.38 GBytes 11.8 Gbits/sec 0 3.10 MBytes [ 5] 1.00-2.00 sec 1.37 GBytes 11.8 Gbits/sec 0 3.10 MBytes [ 5] 2.00-3.00 sec 1.42 GBytes 12.2 Gbits/sec 0 3.10 MBytes [ 5] 3.00-4.00 sec 1.39 GBytes 11.9 Gbits/sec 0 3.10 MBytes [ 5] 4.00-5.00 sec 1.38 GBytes 11.8 Gbits/sec 0 3.10 MBytes [ 5] 5.00-6.00 sec 1.43 GBytes 12.3 Gbits/sec 0 3.10 MBytes [ 5] 6.00-7.00 sec 1.41 GBytes 12.1 Gbits/sec 0 3.10 MBytes [ 5] 7.00-8.00 sec 1.41 GBytes 12.1 Gbits/sec 0 3.10 MBytes [ 5] 8.00-9.00 sec 1.41 GBytes 12.1 Gbits/sec 0 3.10 MBytes [ 5] 9.00-10.00 sec 1.42 GBytes 12.2 Gbits/sec 0 3.10 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 14.0 GBytes 12.0 Gbits/sec 0 sender [ 5] 0.00-10.04 sec 14.0 GBytes 12.0 Gbits/sec receiver iperf Done. ``` **VM-to-VM (through overlay network)** ``` $ iperf3 -c 10.0.6.10 Connecting to host 10.0.6.10, port 5201 [ 5] local 10.0.6.37 port 56710 connected to 10.0.6.10 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 499 MBytes 4.19 Gbits/sec 263 463 KBytes [ 5] 1.00-2.00 sec 483 MBytes 4.05 Gbits/sec 467 367 KBytes [ 5] 2.00-3.00 sec 482 MBytes 4.05 Gbits/sec 491 386 KBytes [ 5] 3.00-4.00 sec 483 MBytes 4.05 Gbits/sec 661 381 KBytes [ 5] 4.00-5.00 sec 472 MBytes 3.95 Gbits/sec 430 391 KBytes [ 5] 5.00-6.00 sec 480 MBytes 4.03 Gbits/sec 474 350 KBytes [ 5] 6.00-7.00 sec 510 MBytes 4.28 Gbits/sec 567 474 KBytes [ 5] 7.00-8.00 sec 521 MBytes 4.37 Gbits/sec 565 387 KBytes [ 5] 8.00-9.00 sec 509 MBytes 4.27 Gbits/sec 632 483 KBytes [ 5] 9.00-10.00 sec 514 MBytes 4.30 Gbits/sec 555 495 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.84 GBytes 4.15 Gbits/sec 5105 sender [ 5] 0.00-10.05 sec 4.84 GBytes 4.14 Gbits/sec receiver iperf Done. ``` As you can see, there's a sharp drop in performance inside the VMs (from ~12 Gbps down to ~4 Gbps), with a lot of retransmissions in 10 seconds. Has anyone encountered a similar issue with OVN and VM networking performance? What could be the potential causes? Any suggestions or debugging tips are welcome. Thanks in advance! Best regards,

5 4

[Neutron] - General thoughts on tenancy network MTU
by Karl Kloppenborg 01 Aug '25

01 Aug '25

Hi Neutrinos, I was discussing with a colleague about our MTU settings within our Openstack regions, generally speaking we have a physnet_global_mtu of 1550 with tenancy networks automatically provisioned as 1500 VXLAN. This ends up with VM’s receiving DHCP MTU settings of 1500 which is fine, however reading about yesterday’s discussion on OVN performance, Nguyen’s recommendation was to enable Jumboframes. This got me thinking, what do you set your tenancy and phynet MTU’s to? Very interested to hear about your experiences with running 9k MTU’s on the VM networks etc, what impact this has on OVS/OVN routers and northbound connectivity being 1500. Thanks, Karl. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-sta…<https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-sta…> This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact(a)resetdata.com.au<mailto:contact@resetdata.com.au>

2 1

(Ceilometer) Openstack - Kolla-Ansible deployment - Ceilometer (Central & compute) Docker Containers are unhealthy
by TRB PREM KUMAR 01 Aug '25

01 Aug '25

Hi, We have deployed an all-in-one OpenStack server for billing. Now ceilometer containers are unhealthy. Please help us to resolve this issue. Reference Output for reference: trb1@openstack-aio-01:~$ docker ps | grep ceil c53fc69ba42b quay.io/openstack.kolla/ceilometer-compute:2024.2-rocky-9 "dumb-init --single-…" About a minute ago Up About a minute *(unhealthy) * ceilometer_compute 9337af646e0f quay.io/openstack.kolla/ceilometer-central:2024.2-rocky-9 "dumb-init --single-…" 30 minutes ago Up About a minute ( *unhealthy)* ceilometer_central 554c4b8ccd86 quay.io/openstack.kolla/ceilometer-notification:2024.2-rocky-9 "dumb-init --single-…" 4 days ago Up About a minute (healthy) ceilometer_notification *** Logs files are attached for reference. Regards, TRB.

3 6

[scientific-sig][help] Research about private cloud
by Winicius Allan 31 Jul '25

31 Jul '25

Hi all! I'm about to write my final paper, and I want to research something related to OpenStack and private clouds. If someone wants to take the time to discuss the problems in your environment, I'd be very glad!What I had on my mind is: - There is something that could improve utilization and performance when the user is not charged about their use on a private cloud. - There is any process scheduling algorithm that could be used on specific workloads instead of Linux default scheduler that can improve application performance? I was thinking about using eBPF here. Best regards.

2 1

[cinder] AttributeError: 'Volume' object has no attribute 'use_quota'
by Kees Meijs | Nefos 31 Jul '25

31 Jul '25

Hello Stackers, It seems we're hitting a bug within the Cinder Volume service. The problem presents itself when creating a Ceph RBD backed Volume using an originating snapshot. > 2025-07-31 19:02:54.896 3602390 WARNING cinder.volume.manager > [req-a476ac04-ee6c-4335-8fca-e606616a5959 > eccdb27249254d9d9fb73570f5de8877 d678ba8e1dee4f7484555580d0a61562 - - > -] Task > 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' > (03c06b47-7df3-4427-8ad0-61f9a97d0159) transitioned into state > 'FAILURE' from state 'RUNNING' > 5 predecessors (most recent first): > </snip (maybe the Atoms are needed and if so I could redact those)> > |__Flow 'volume_create_manager': AttributeError: 'Volume' > object has no attribute 'use_quota' > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager Traceback > (most recent call last): > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager File > "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", > line 53, in _execute_task > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager result = > task.execute(**arguments) > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager File > "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", > line 1158, in execute > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager > model_update = self._create_from_snapshot(context, volume, > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager File > "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", > line 461, in _create_from_snapshot > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager > model_update = self.driver.create_volume_from_snapshot(volume, > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager File > "/usr/lib/python3/dist-packages/cinder/volume/drivers/rbd.py", line > 1069, in create_volume_from_snapshot > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager if > int(volume.size): > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager > AttributeError: 'Volume' object has no attribute 'use_quota' > 2025-07-31 19:02:54.896 3602390 ERROR cinder.volume.manager Yes, this reflects a non-current version, to be precise Cinder 17.4. However, does this look familiar to someone, maybe? Thanks! Cheers, Kees

1 1

[Neutron] ML2: Can Operators Safely Extend binding:profile with Custom Keys?
by cxue22＠bloomberg.net 31 Jul '25

31 Jul '25

Hi all, I'm working on a Neutron customization to expose and manage data in the `binding:profile` field stored in the `ml2_port_bindings` table, and I have a few questions around operator and third-party plugin compatibility — particularly in reference to the blueprint: https://blueprints.launchpad.net/neutron/+spec/ml2-binding-profile As I understand it, the `binding:profile` is represented as a string column containing a JSON dictionary that can be passed by the compute agent and interpreted by ML2 mechanism drivers during port binding. My intent is to: - Create a custom API endpoint that allows reading and updating specific key/value pairs in this JSON dictionary. - Enable third-party mechanism drivers and operators to add custom fields to this profile. My main question is: **Are operators and third-party ML2 drivers free to add arbitrary key/value pairs to the `binding:profile` dictionary, and can those be expected to persist across Neutron upgrades and database migrations?** Additionally: - Are there any best practices or expectations around namespacing or schema validation of custom fields in `binding:profile`? - Are there known caveats or upgrade scenarios where such data may be truncated, filtered, or overwritten? I'd appreciate any guidance from the Neutron community on how this field is intended to be extended or preserved, and whether such customization is supported in the long term. Thanks in advance! Best regards, Chang Xue Bloomberg Engineering

3 3

[openstack][magnum]Cannot upgrade clusters
by Nguyễn Hữu Khôi 31 Jul '25

31 Jul '25

Hello. I cannot upgrade clusters via magnum_cluster_api. These are log from magnum_api: https://anotepad.com/notes/c9h8bd55 OPS: 2025.1 OS: Ubuntu 24.04 magnum_cluster_api: v0.30.0 Would you give some ideas on it? Thank you! Nguyen Huu Khoi

1 1

[Neutron][VPNaaS] Multiple tunnels per IPSec Service
by Shubham Yadav 31 Jul '25

31 Jul '25

Hello OpenStack Community, I'm working with OpenStack VPNaaS (using strongSwan as the backend driver) and have a question about tunnel configuration capabilities. *Current Setup:* OpenStack VPNaaS with strongSwan driver (StrongSwanDriver) Single VPN Service (gateway) with multiple IPSec Site Connections Each connection targets different remote peers with distinct subnets *Configuration Details:* VPN Service: Connected to a single router Multiple IPSec Site Connections under the same VPN Service: Connection 1: To MY office (10.10.100.0/24) via peer 68.x.y.34 Connection 2: To Rackspace (172.24.60.0/22) via peer 184.n.p.53 Shared IKE and IPSec policies across connections Local endpoint groups covering multiple subnets (192.168.0.0/24, 10.43.0.0/16) *Question:*Is it supported/recommended to have multiple IPSec Site Connections under a single VPN Service? I understand this creates multiple tunnels from the same OpenStack router to different remote peers. *Current Issue:*One of my connections shows "PENDING_CREATE" status while the other is "ACTIVE". I'm trying to determine if this is: A configuration issue on my end A limitation of having multiple tunnels per VPN service A strongSwan-specific behavior in the OpenStack context *Specific Questions:* Does VPNaaS officially support multiple IPSec connections per VPN Service? Are there any known limitations or best practices when using multiple tunnels? Should each remote site have its own dedicated VPN Service instead? I've checked the documentation but couldn't find clear guidance on multi-tunnel scenarios. Any insights from the community would be greatly appreciated. *Environment:* OpenStack version: 2024.2 Dalmatian Neutron VPNaaS with strongSwan backend Kubernetes-deployed OpenStack (OpenStack-Helm) Thank you for your time and assistance! Best regards, Shubham

1 0

[Neutron][VPNaaS] Multiple tunnels per IPSec Service
by Shubham Yadav 31 Jul '25

31 Jul '25

Hello OpenStack Community, I'm working with OpenStack VPNaaS (using strongSwan as the backend driver) and have a question about tunnel configuration capabilities. Current Setup: OpenStack VPNaaS with strongSwan driver (StrongSwanDriver) Single VPN Service (gateway) with multiple IPSec Site Connections Each connection targets different remote peers with distinct subnets Configuration Details: VPN Service: Connected to a single router Multiple IPSec Site Connections under the same VPN Service: Connection 1: To MY office (10.10.100.0/24) via peer 68.x.y.34 Connection 2: To Rackspace (172.24.60.0/22) via peer 184.n.p.53 Shared IKE and IPSec policies across connections Local endpoint groups covering multiple subnets (192.168.0.0/24, 10.43.0.0/16) Question: Is it supported/recommended to have multiple IPSec Site Connections under a single VPN Service? I understand this creates multiple tunnels from the same OpenStack router to different remote peers. Current Issue: One of my connections shows "PENDING_CREATE" status while the other is "ACTIVE". I'm trying to determine if this is: A configuration issue on my end A limitation of having multiple tunnels per VPN service A strongSwan-specific behavior in the OpenStack context Specific Questions: Does VPNaaS officially support multiple IPSec connections per VPN Service? Are there any known limitations or best practices when using multiple tunnels? Should each remote site have its own dedicated VPN Service instead? I've checked the documentation but couldn't find clear guidance on multi-tunnel scenarios. Any insights from the community would be greatly appreciated. Environment: OpenStack version: 2024.2 Dalmatian Neutron VPNaaS with strongSwan backend Kubernetes-deployed OpenStack (OpenStack-Helm) Thank you for your time and assistance! Best regards, Shubham

1 0

[Ovs-dpdk]Issue with ovs-dpdk deployment in Kolla-Ansible (zed)
by samira_darvishnia＠yahoo.com 30 Jul '25

30 Jul '25

Hi all, I'm trying to deploy OVS-DPDK using Kolla-Ansible Zed, and I'm running into a blocking issue during deployment. I would really appreciate your insights or suggestions. Problem Summary: During the deployment of ovs-dpdk, the container ovsdpdk-vswitchd fails to start, with the following error in its logs: + echo 'Running command: '\''/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile'\''' + exec /usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile Running command: '/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile' ovs-vswitchd: invalid option -- 'w' ovs-vswitchd: invalid option -- 'w' EAL: FATAL: Invalid 'command line' arguments. 2025-07-30T13:44:59Z|00017|dpdk|EMER|Unable to initialize DPDK: Invalid argument ovs-vswitchd: Cannot init EAL (Invalid argument) This appears to be related to an incorrect parsing of arguments for ovs-vswitchd. What I’ve tried: I already applied this patch (as referenced in Gerrit): https://review.opendev.org/c/openstack/kolla-ansible/+/913653/1/ansible/rol… However, the issue still persists. Additional context: • Deployment hangs on the task: name: Wait for dpdk tunnel ip It waits for 5 minutes and then fails. • The parameter dpdk_tunnel_interface_address is already set in my global.yml. • I’ve configured and bound NICs to vfio-pci. • Environment info: • OpenStack Zed • Kolla-Ansible Zed • dpdk_tunnel_interface_address is set • OVS-DPDK is containerized via Kolla Has anyone seen this issue before or can suggest further steps? Any insight on how to correctly pass EAL arguments or what else might be misconfigured would be very helpful. Thanks in advance, Samira

1 0