[Openstack] need help understanding neutron/OVS
Manuel Sopena Ballesteros
manuel.sb at garvan.org.au
Tue Dec 19 01:38:40 UTC 2017
Dear Openstack community,
Openstack environment:
2 compute nodes each has:
* 1x AMD Opteron(tm) Processor 6282 SE (56 cpus)
* 512GB RAM
* 1x dual port 25Gbps Mellanox connect-4 XOR bond
Physical network:
* Compute nodes connected through a mellanox 2700 switch (100Gbps)
* MTU 9000
Openstack network:
* OVS + VxLan
* Vxlan offloading setup on physical nics
* MTU on vms 8950
Compute nodes nics:
[root at hercules-23 ~]# ip a
...
6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.32.23/16 brd 10.0.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::3826:9daa:ef1b:43b0/64 scope link
valid_lft forever preferred_lft forever
...
VM nics:
[centos at centos2 ~]$ ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:a3:d4:85 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0
valid_lft 57885sec preferred_lft 57885sec
inet6 fe80::f816:3eff:fea3:d485/64 scope link
valid_lft forever preferred_lft forever
host to host bandwidth using muttcp (1 thread):
[root at hercules-22 ~]# nuttcp -l65536 -fparse 10.0.32.23
megabytes=19844.1759 real_seconds=10.00 rate_Mbps=16643.4189 tx_cpu=61 rx_cpu=99 retrans=0 rtt_ms=0.28
VMTP Output:
==================
Total Scenarios: 29
Passed Scenarios: 17 [100.00%]
Failed Scenarios: 0 [0.00%]
Skipped Scenarios: 12
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| Scenario | Scenario Name | Functional Status | Data |
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| 1.1 | Same Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '7613130', 'rtt_ms': '0.463333'} |
| 1.2 | Same Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 59954, 'loss_rate': 0.02}, 1024: {'tp_kbps': 444264, |
| | | | 'loss_rate': 0.05}, 8192: {'tp_kbps': 2877674, 'loss_rate': 0.0}} |
| 1.3 | Same Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.376/0.285/0.439/0.047', |
| | | | '64-byte': '0.617/0.361/1.513/0.388', '1500-byte': '0.369/0.322/0.452/0.048'}} |
| 1.4 | Same Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
| 2.1 | Same Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6864266', 'rtt_ms': '0.653333'} |
| 2.2 | Same Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 80955, 'loss_rate': 0.46}, 1024: {'tp_kbps': 166186, |
| | | | 'loss_rate': 36.2}, 8192: {'tp_kbps': 2189389, 'loss_rate': 0.05}} |
| 2.3 | Same Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.484/0.315/0.561/0.068', |
| | | | '64-byte': '0.719/0.421/2.714/0.666', '1500-byte': '0.465/0.358/0.553/0.055'}} |
| 2.4 | Same Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
| 3.1 | Different Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '5207719', 'rtt_ms': '0.896667'} |
| 3.2 | Different Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 76569, 'loss_rate': 0.9}, 1024: {'tp_kbps': 571847, |
| | | | 'loss_rate': 2.75}, 8192: {'tp_kbps': 1855166, 'loss_rate': 0.0}} |
| 3.3 | Different Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.859/0.658/1.103/0.133', |
| | | | '64-byte': '0.976/0.576/1.946/0.361', '1500-byte': '0.824/0.644/1.082/0.136'}} |
| 3.4 | Different Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
| 4.1 | Different Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6730928', 'rtt_ms': '1.37667'} |
| 4.2 | Different Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 31131, 'loss_rate': 4.84}, 1024: {'tp_kbps': 579859, |
| | | | 'loss_rate': 4.35}, 8192: {'tp_kbps': 1601092, 'loss_rate': 0.18}} |
| 4.3 | Different Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.915/0.740/1.138/0.144', |
| | | | '64-byte': '1.134/0.713/2.210/0.539', '1500-byte': '0.980/0.719/1.188/0.131'}} |
| 4.4 | Different Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
| 5.1 | Different Network, Floating IP, Intra-node, TCP | PASSED | {'tp_kbps': '5406554', 'rtt_ms': '0.836667'} |
| 5.2 | Different Network, Floating IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 71084, 'loss_rate': 4.26}, 1024: {'tp_kbps': 507900, |
| | | | 'loss_rate': 15.96}, 8192: {'tp_kbps': 2011414, 'loss_rate': 3.72}} |
| 5.3 | Different Network, Floating IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.902/0.760/1.007/0.073', |
| | | | '64-byte': '0.875/0.661/0.967/0.104', '1500-byte': '0.889/0.754/0.988/0.082'}} |
| 5.4 | Different Network, Floating IP, Intra-node, Multicast | SKIPPED | {} |
| 6.1 | Different Network, Floating IP, Inter-node, TCP | PASSED | {'tp_kbps': '6770995', 'rtt_ms': '0.973333'} |
| 6.2 | Different Network, Floating IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 81812, 'loss_rate': 2.25}, 1024: {'tp_kbps': 617485, |
| | | | 'loss_rate': 3.27}, 8192: {'tp_kbps': 1295351, 'loss_rate': 0.02}} |
| 6.3 | Different Network, Floating IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '1.028/0.920/1.254/0.105', |
| | | | '64-byte': '1.043/0.807/1.227/0.137', '1500-byte': '0.988/0.759/1.194/0.146'}} |
| 6.4 | Different Network, Floating IP, Inter-node, Multicast | SKIPPED | {} |
| 7.1 | Native Throughput, TCP | SKIPPED | {} |
| 7.2 | Native Throughput, UDP | SKIPPED | {} |
| 7.3 | Native Throughput, ICMP | SKIPPED | {} |
| 7.4 | Native Throughput, Multicast | SKIPPED | {} |
| 8.1 | VM to Host Uploading | SKIPPED | {} |
| 8.2 | VM to Host Downloading | SKIPPED | {} |
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
My concern is about east-west bandwidth:
Inter-host and intra-subnet (vmtp Scenario 2.1): I am getting ~6.8Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
intra-host and intra-subnet (vmtp Scenario 1.1): In this example traffic remains inside the compute node and no tunneling is required. I am getting ~7.6Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
NOTE: I am not running anything except these tests on the machines so cpu usage is quite low.
NOTE2: I know vmtp uses
I would like to understand what is stopping my vms to get full network bandwidth. Is this related with an OVS configuration? ML2? Other part of neutron? Something else?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171219/670c072c/attachment.html>
More information about the Openstack
mailing list