[Openstack] poor bandwidth across instances running on same host

Tomáš Vondra vondra at homeatcloud.cz
Tue Apr 18 12:12:04 UTC 2017


Sorry to shatter you expectations, but those numbers are perfectly OK.

I was testing on HPE DL380 gen9 with Intel Xeon E5-2630v3 <https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2630+v3+%40+2.40GHz&id=2386&cpuCount=2>  and I got these speeds between two KVM VMs on the same host using netperf

28186 Mb/s with linux bridge

18552 Mb/s with OpenVSwitch with the full Neutron setup with iptables.

 

How much would you like to achieve? I got 38686 on dev lo on the physical server and 47894 on a VM. You could turn to OVS with DPDK as the data path, bu I doubt it will do much. SR-IOV might, but I never tried any of this. I’m satisfied with the speed for my purposes.

Tomas from Homeatcloud

 

From: Manuel Sopena Ballesteros [mailto:manuel.sb at garvan.org.au] 
Sent: Tuesday, April 18, 2017 9:11 AM
To: openstack at lists.openstack.org
Subject: [Openstack] poor bandwidth across instances running on same host

 

Hi all,

 

I created 2 instances on the same compute node and tested the bandwidth between them, surprisingly iperf tells me I got 16.1Gbits/sec only. Then I changed the firewall from hybrid iptables to ovs, the bandwidth improved a little bit to 17.5Gbits/sec but still far from expected. 

 

Ml2_config.ini config file

 

[root at nova-compute ~]# docker exec -t neutron_openvswitch_agent vi /var/lib/kolla/config_files/ml2_config.ini
network_vlan_ranges =
 
[ml2_type_flat]
flat_networks = physnet1
 
[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1
 
[securitygroup]
firewall_driver = openvswitch
#firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
 
[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true
 
[ovs]
bridge_mappings = physnet1:br-ex
ovsdb_connection = tcp:129.94.72.54:6640
local_ip = 10.1.0.12

 

 

 

ovs config

 

[root at nova-compute ~]# docker exec openvswitch_vswitchd ovs-vsctl show
306d62c4-8e35-45e0-838e-53ebe81f1d06
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "eno50336512"
            Interface "eno50336512"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a01000b"
            Interface "vxlan-0a01000b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.1.0.12", out_key=flow, remote_ip="10.1.0.11"}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tapa26ee521-3b"
            tag: 2
            Interface "tapa26ee521-3b"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "tap1f76851b-ea"
            tag: 2
            Interface "tap1f76851b-ea"
 
 
 

Iperf results

 
[centos at centos7 ~]$ iperf -c 192.168.1.105
------------------------------------------------------------
Client connecting to 192.168.1.105, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.101 port 48522 connected with 192.168.1.105 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  20.3 GBytes  17.5 Gbits/sec

 

 

 

Ovs info

 

[root at nova-compute ~]# docker exec openvswitch_vswitchd modinfo openvswitch

filename:       /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko

license:        GPL

description:    Open vSwitch switching datapath

rhelversion:    7.3

srcversion:     B31AE95554C9D9A0067F935

depends:        nf_conntrack,nf_nat,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6

intree:         Y

vermagic:       3.10.0-514.el7.x86_64 SMP mod_unload modversions

signer:         CentOS Linux kernel signing key

sig_key:        D4:88:63:A7:C1:6F:CC:27:41:23:E6:29:8F:74:F0:57:AF:19:FC:54

sig_hashalgo:   sha256

 

 

As far as I know the communication is VM ßOVSàVM and the linux bridge is not involved.

 

What could be throttling the network traffic and what can I do to improve performance?

 

Thank you very much

 

Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research 
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au

 

NOTICE

Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170418/ecf38a5d/attachment.html>


More information about the Openstack mailing list