[Openstack] need help understanding neutron/OVS
Manuel Sopena Ballesteros
manuel.sb at garvan.org.au
Tue Dec 19 04:19:59 UTC 2017
Ok,
This is it:
(neutron-openvswitch-agent)[root at hercules-23 /]# grep -i mtu /etc/neutron/plugins/ml2/openvswitch_agent.ini
# MTU size of veth interfaces (integer value)
#veth_mtu = 9000
Thank you
Manuel
From: Remo Mattei [mailto:Remo at italy1.com]
Sent: Tuesday, December 19, 2017 3:09 PM
To: Manuel Sopena Ballesteros
Cc: openstack at lists.openstack.org
Subject: Re: [Openstack] need help understanding neutron/OVS
I think you are looking at the wrong file.. if you are using ovs you need to look at that… example:
[root at os-poc-controller-0 ml2]# grep -i mtu openvswitch_agent.ini
# MTU size of veth interfaces (integer value)
#veth_mtu = 9000
On Dec 18, 2017, at 20:06, Manuel Sopena Ballesteros <manuel.sb at garvan.org.au<mailto:manuel.sb at garvan.org.au>> wrote:
Yes,
This is my neutron ml2 config on the compute node
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
path_mtu = 9000
mechanism_drivers = openvswitch,l2population
extension_drivers = port_security
[ml2_type_vlan]
network_vlan_ranges =
[ml2_type_flat]
flat_networks = physnet1
[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
[agent]
tunnel_types = vxlan
l2_population = true
arp_responder = true
[ovs]
ovsdb_connection = tcp:127.0.0.1:6640
local_ip = 10.0.32.23
thank you
Manuel
From: Remo Mattei [mailto:remo at italy1.com]
Sent: Tuesday, December 19, 2017 12:56 PM
To: Manuel Sopena Ballesteros
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] need help understanding neutron/OVS
Did you change the mtu on neutron config?
Sent from my iPad
On Dec 18, 2017, at 5:38 PM, Manuel Sopena Ballesteros <manuel.sb at garvan.org.au<mailto:manuel.sb at garvan.org.au>> wrote:
Dear Openstack community,
Openstack environment:
2 compute nodes each has:
• 1x AMD Opteron(tm) Processor 6282 SE (56 cpus)
• 512GB RAM
• 1x dual port 25Gbps Mellanox connect-4 XOR bond
Physical network:
• Compute nodes connected through a mellanox 2700 switch (100Gbps)
• MTU 9000
Openstack network:
• OVS + VxLan
• Vxlan offloading setup on physical nics
• MTU on vms 8950
Compute nodes nics:
[root at hercules-23 ~]# ip a
…
6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.32.23/16 brd 10.0.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::3826:9daa:ef1b:43b0/64 scope link
valid_lft forever preferred_lft forever
…
VM nics:
[centos at centos2 ~]$ ip a
…
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:a3:d4:85 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0
valid_lft 57885sec preferred_lft 57885sec
inet6 fe80::f816:3eff:fea3:d485/64 scope link
valid_lft forever preferred_lft forever
host to host bandwidth using muttcp (1 thread):
[root at hercules-22 ~]# nuttcp -l65536 -fparse 10.0.32.23
megabytes=19844.1759 real_seconds=10.00 rate_Mbps=16643.4189 tx_cpu=61 rx_cpu=99 retrans=0 rtt_ms=0.28
VMTP Output:
==================
Total Scenarios: 29
Passed Scenarios: 17 [100.00%]
Failed Scenarios: 0 [0.00%]
Skipped Scenarios: 12
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| Scenario | Scenario Name | Functional Status | Data |
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| 1.1 | Same Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '7613130', 'rtt_ms': '0.463333'} |
| 1.2 | Same Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 59954, 'loss_rate': 0.02}, 1024: {'tp_kbps': 444264, |
| | | | 'loss_rate': 0.05}, 8192: {'tp_kbps': 2877674, 'loss_rate': 0.0}} |
| 1.3 | Same Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.376/0.285/0.439/0.047', |
| | | | '64-byte': '0.617/0.361/1.513/0.388', '1500-byte': '0.369/0.322/0.452/0.048'}} |
| 1.4 | Same Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
| 2.1 | Same Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6864266', 'rtt_ms': '0.653333'} |
| 2.2 | Same Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 80955, 'loss_rate': 0.46}, 1024: {'tp_kbps': 166186, |
| | | | 'loss_rate': 36.2}, 8192: {'tp_kbps': 2189389, 'loss_rate': 0.05}} |
| 2.3 | Same Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.484/0.315/0.561/0.068', |
| | | | '64-byte': '0.719/0.421/2.714/0.666', '1500-byte': '0.465/0.358/0.553/0.055'}} |
| 2.4 | Same Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
| 3.1 | Different Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '5207719', 'rtt_ms': '0.896667'} |
| 3.2 | Different Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 76569, 'loss_rate': 0.9}, 1024: {'tp_kbps': 571847, |
| | | | 'loss_rate': 2.75}, 8192: {'tp_kbps': 1855166, 'loss_rate': 0.0}} |
| 3.3 | Different Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.859/0.658/1.103/0.133', |
| | | | '64-byte': '0.976/0.576/1.946/0.361', '1500-byte': '0.824/0.644/1.082/0.136'}} |
| 3.4 | Different Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
| 4.1 | Different Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6730928', 'rtt_ms': '1.37667'} |
| 4.2 | Different Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 31131, 'loss_rate': 4.84}, 1024: {'tp_kbps': 579859, |
| | | | 'loss_rate': 4.35}, 8192: {'tp_kbps': 1601092, 'loss_rate': 0.18}} |
| 4.3 | Different Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.915/0.740/1.138/0.144', |
| | | | '64-byte': '1.134/0.713/2.210/0.539', '1500-byte': '0.980/0.719/1.188/0.131'}} |
| 4.4 | Different Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
| 5.1 | Different Network, Floating IP, Intra-node, TCP | PASSED | {'tp_kbps': '5406554', 'rtt_ms': '0.836667'} |
| 5.2 | Different Network, Floating IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 71084, 'loss_rate': 4.26}, 1024: {'tp_kbps': 507900, |
| | | | 'loss_rate': 15.96}, 8192: {'tp_kbps': 2011414, 'loss_rate': 3.72}} |
| 5.3 | Different Network, Floating IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.902/0.760/1.007/0.073', |
| | | | '64-byte': '0.875/0.661/0.967/0.104', '1500-byte': '0.889/0.754/0.988/0.082'}} |
| 5.4 | Different Network, Floating IP, Intra-node, Multicast | SKIPPED | {} |
| 6.1 | Different Network, Floating IP, Inter-node, TCP | PASSED | {'tp_kbps': '6770995', 'rtt_ms': '0.973333'} |
| 6.2 | Different Network, Floating IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 81812, 'loss_rate': 2.25}, 1024: {'tp_kbps': 617485, |
| | | | 'loss_rate': 3.27}, 8192: {'tp_kbps': 1295351, 'loss_rate': 0.02}} |
| 6.3 | Different Network, Floating IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '1.028/0.920/1.254/0.105', |
| | | | '64-byte': '1.043/0.807/1.227/0.137', '1500-byte': '0.988/0.759/1.194/0.146'}} |
| 6.4 | Different Network, Floating IP, Inter-node, Multicast | SKIPPED | {} |
| 7.1 | Native Throughput, TCP | SKIPPED | {} |
| 7.2 | Native Throughput, UDP | SKIPPED | {} |
| 7.3 | Native Throughput, ICMP | SKIPPED | {} |
| 7.4 | Native Throughput, Multicast | SKIPPED | {} |
| 8.1 | VM to Host Uploading | SKIPPED | {} |
| 8.2 | VM to Host Downloading | SKIPPED | {} |
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
My concern is about east-west bandwidth:
Inter-host and intra-subnet (vmtp Scenario 2.1): I am getting ~6.8Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
intra-host and intra-subnet (vmtp Scenario 1.1): In this example traffic remains inside the compute node and no tunneling is required. I am getting ~7.6Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
NOTE: I am not running anything except these tests on the machines so cpu usage is quite low.
NOTE2: I know vmtp uses
I would like to understand what is stopping my vms to get full network bandwidth. Is this related with an OVS configuration? ML2? Other part of neutron? Something else?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au<mailto:manuel.sb at garvan.org.au>
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
[https://master.mailbutler.io/tracking/70F490E0-E398-4120-A92A-6CF45ECC075B]
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171219/1f09d324/attachment.html>
More information about the Openstack
mailing list