[Openstack] poor bandwidth across instances running on same host
Manuel Sopena Ballesteros
manuel.sb at garvan.org.au
Wed Apr 19 01:10:14 UTC 2017
Hi Tomas,
I would expect bigger bandwidth I was in contact with SamYaple from the #openstack IRC channel who got 40Gigs/sec in his lab. The main difference between our environments:
Me:
· DPDK: NO
· SR-IOV: NO
· OS: Centos 7.3 (kernel 3.10)
SamYaple:
· DPDK: YES
· SR-IOV: YES
· OS: Ubuntu (kernel 4.1)
Do you think the kernel may have a huge impact in the ovs performance?
Thank you very much
Manuel
From: Tomáš Vondra [mailto:vondra at homeatcloud.cz]
Sent: Tuesday, April 18, 2017 10:12 PM
To: Manuel Sopena Ballesteros
Cc: openstack at lists.openstack.org
Subject: RE: [Openstack] poor bandwidth across instances running on same host
Sorry to shatter you expectations, but those numbers are perfectly OK.
I was testing on HPE DL380 gen9 with Intel Xeon E5-2630v3<https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2630+v3+%40+2.40GHz&id=2386&cpuCount=2> and I got these speeds between two KVM VMs on the same host using netperf
28186 Mb/s with linux bridge
18552 Mb/s with OpenVSwitch with the full Neutron setup with iptables.
How much would you like to achieve? I got 38686 on dev lo on the physical server and 47894 on a VM. You could turn to OVS with DPDK as the data path, bu I doubt it will do much. SR-IOV might, but I never tried any of this. I’m satisfied with the speed for my purposes.
Tomas from Homeatcloud
From: Manuel Sopena Ballesteros [mailto:manuel.sb at garvan.org.au]
Sent: Tuesday, April 18, 2017 9:11 AM
To: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: [Openstack] poor bandwidth across instances running on same host
Hi all,
I created 2 instances on the same compute node and tested the bandwidth between them, surprisingly iperf tells me I got 16.1Gbits/sec only. Then I changed the firewall from hybrid iptables to ovs, the bandwidth improved a little bit to 17.5Gbits/sec but still far from expected.
Ml2_config.ini config file
[root at nova-compute ~]# docker exec -t neutron_openvswitch_agent vi /var/lib/kolla/config_files/ml2_config.ini
network_vlan_ranges =
[ml2_type_flat]
flat_networks = physnet1
[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1
[securitygroup]
firewall_driver = openvswitch
#firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true
[ovs]
bridge_mappings = physnet1:br-ex
ovsdb_connection = tcp:129.94.72.54:6640
local_ip = 10.1.0.12
ovs config
[root at nova-compute ~]# docker exec openvswitch_vswitchd ovs-vsctl show
306d62c4-8e35-45e0-838e-53ebe81f1d06
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "eno50336512"
Interface "eno50336512"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port br-ex
Interface br-ex
type: internal
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-0a01000b"
Interface "vxlan-0a01000b"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.1.0.12", out_key=flow, remote_ip="10.1.0.11"}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "tapa26ee521-3b"
tag: 2
Interface "tapa26ee521-3b"
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "tap1f76851b-ea"
tag: 2
Interface "tap1f76851b-ea"
Iperf results
[centos at centos7 ~]$ iperf -c 192.168.1.105
------------------------------------------------------------
Client connecting to 192.168.1.105, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.101 port 48522 connected with 192.168.1.105 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 20.3 GBytes 17.5 Gbits/sec
Ovs info
[root at nova-compute ~]# docker exec openvswitch_vswitchd modinfo openvswitch
filename: /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
license: GPL
description: Open vSwitch switching datapath
rhelversion: 7.3
srcversion: B31AE95554C9D9A0067F935
depends: nf_conntrack,nf_nat,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6
intree: Y
vermagic: 3.10.0-514.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: D4:88:63:A7:C1:6F:CC:27:41:23:E6:29:8F:74:F0:57:AF:19:FC:54
sig_hashalgo: sha256
As far as I know the communication is VM <--OVS-->VM and the linux bridge is not involved.
What could be throttling the network traffic and what can I do to improve performance?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au<mailto:manuel.sb at garvan.org.au>
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170419/36c27076/attachment.html>
More information about the Openstack
mailing list