[Openstack] need help understanding neutron/OVS
Remo Mattei
remo at italy1.com
Tue Dec 19 01:55:38 UTC 2017
Did you change the mtu on neutron config?
Sent from my iPad
> On Dec 18, 2017, at 5:38 PM, Manuel Sopena Ballesteros <manuel.sb at garvan.org.au> wrote:
>
> Dear Openstack community,
>
> Openstack environment:
>
> 2 compute nodes each has:
> · 1x AMD Opteron(tm) Processor 6282 SE (56 cpus)
> · 512GB RAM
> · 1x dual port 25Gbps Mellanox connect-4 XOR bond
>
> Physical network:
> · Compute nodes connected through a mellanox 2700 switch (100Gbps)
> · MTU 9000
>
> Openstack network:
> · OVS + VxLan
> · Vxlan offloading setup on physical nics
> · MTU on vms 8950
>
> Compute nodes nics:
>
> [root at hercules-23 ~]# ip a
> …
> 6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
> link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
> 7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000
> link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
> 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP qlen 1000
> link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
> inet 10.0.32.23/16 brd 10.0.255.255 scope global bond0
> valid_lft forever preferred_lft forever
> inet6 fe80::3826:9daa:ef1b:43b0/64 scope link
> valid_lft forever preferred_lft forever
> …
>
>
> VM nics:
>
> [centos at centos2 ~]$ ip a
> …
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP qlen 1000
> link/ether fa:16:3e:a3:d4:85 brd ff:ff:ff:ff:ff:ff
> inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0
> valid_lft 57885sec preferred_lft 57885sec
> inet6 fe80::f816:3eff:fea3:d485/64 scope link
> valid_lft forever preferred_lft forever
>
>
> host to host bandwidth using muttcp (1 thread):
>
> [root at hercules-22 ~]# nuttcp -l65536 -fparse 10.0.32.23
> megabytes=19844.1759 real_seconds=10.00 rate_Mbps=16643.4189 tx_cpu=61 rx_cpu=99 retrans=0 rtt_ms=0.28
>
> VMTP Output:
>
> ==================
> Total Scenarios: 29
> Passed Scenarios: 17 [100.00%]
> Failed Scenarios: 0 [0.00%]
> Skipped Scenarios: 12
> +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
> | Scenario | Scenario Name | Functional Status | Data |
> +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
> | 1.1 | Same Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '7613130', 'rtt_ms': '0.463333'} |
> | 1.2 | Same Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 59954, 'loss_rate': 0.02}, 1024: {'tp_kbps': 444264, |
> | | | | 'loss_rate': 0.05}, 8192: {'tp_kbps': 2877674, 'loss_rate': 0.0}} |
> | 1.3 | Same Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.376/0.285/0.439/0.047', |
> | | | | '64-byte': '0.617/0.361/1.513/0.388', '1500-byte': '0.369/0.322/0.452/0.048'}} |
> | 1.4 | Same Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
> | 2.1 | Same Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6864266', 'rtt_ms': '0.653333'} |
> | 2.2 | Same Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 80955, 'loss_rate': 0.46}, 1024: {'tp_kbps': 166186, |
> | | | | 'loss_rate': 36.2}, 8192: {'tp_kbps': 2189389, 'loss_rate': 0.05}} |
> | 2.3 | Same Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.484/0.315/0.561/0.068', |
> | | | | '64-byte': '0.719/0.421/2.714/0.666', '1500-byte': '0.465/0.358/0.553/0.055'}} |
> | 2.4 | Same Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
> | 3.1 | Different Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '5207719', 'rtt_ms': '0.896667'} |
> | 3.2 | Different Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 76569, 'loss_rate': 0.9}, 1024: {'tp_kbps': 571847, |
> | | | | 'loss_rate': 2.75}, 8192: {'tp_kbps': 1855166, 'loss_rate': 0.0}} |
> | 3.3 | Different Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.859/0.658/1.103/0.133', |
> | | | | '64-byte': '0.976/0.576/1.946/0.361', '1500-byte': '0.824/0.644/1.082/0.136'}} |
> | 3.4 | Different Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} |
> | 4.1 | Different Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6730928', 'rtt_ms': '1.37667'} |
> | 4.2 | Different Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 31131, 'loss_rate': 4.84}, 1024: {'tp_kbps': 579859, |
> | | | | 'loss_rate': 4.35}, 8192: {'tp_kbps': 1601092, 'loss_rate': 0.18}} |
> | 4.3 | Different Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.915/0.740/1.138/0.144', |
> | | | | '64-byte': '1.134/0.713/2.210/0.539', '1500-byte': '0.980/0.719/1.188/0.131'}} |
> | 4.4 | Different Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} |
> | 5.1 | Different Network, Floating IP, Intra-node, TCP | PASSED | {'tp_kbps': '5406554', 'rtt_ms': '0.836667'} |
> | 5.2 | Different Network, Floating IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 71084, 'loss_rate': 4.26}, 1024: {'tp_kbps': 507900, |
> | | | | 'loss_rate': 15.96}, 8192: {'tp_kbps': 2011414, 'loss_rate': 3.72}} |
> | 5.3 | Different Network, Floating IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.902/0.760/1.007/0.073', |
> | | | | '64-byte': '0.875/0.661/0.967/0.104', '1500-byte': '0.889/0.754/0.988/0.082'}} |
> | 5.4 | Different Network, Floating IP, Intra-node, Multicast | SKIPPED | {} |
> | 6.1 | Different Network, Floating IP, Inter-node, TCP | PASSED | {'tp_kbps': '6770995', 'rtt_ms': '0.973333'} |
> | 6.2 | Different Network, Floating IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 81812, 'loss_rate': 2.25}, 1024: {'tp_kbps': 617485, |
> | | | | 'loss_rate': 3.27}, 8192: {'tp_kbps': 1295351, 'loss_rate': 0.02}} |
> | 6.3 | Different Network, Floating IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '1.028/0.920/1.254/0.105', |
> | | | | '64-byte': '1.043/0.807/1.227/0.137', '1500-byte': '0.988/0.759/1.194/0.146'}} |
> | 6.4 | Different Network, Floating IP, Inter-node, Multicast | SKIPPED | {} |
> | 7.1 | Native Throughput, TCP | SKIPPED | {} |
> | 7.2 | Native Throughput, UDP | SKIPPED | {} |
> | 7.3 | Native Throughput, ICMP | SKIPPED | {} |
> | 7.4 | Native Throughput, Multicast | SKIPPED | {} |
> | 8.1 | VM to Host Uploading | SKIPPED | {} |
> | 8.2 | VM to Host Downloading | SKIPPED | {} |
> +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
>
>
> My concern is about east-west bandwidth:
>
> Inter-host and intra-subnet (vmtp Scenario 2.1): I am getting ~6.8Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
> intra-host and intra-subnet (vmtp Scenario 1.1): In this example traffic remains inside the compute node and no tunneling is required. I am getting ~7.6Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts
>
> NOTE: I am not running anything except these tests on the machines so cpu usage is quite low.
> NOTE2: I know vmtp uses
>
> I would like to understand what is stopping my vms to get full network bandwidth. Is this related with an OVS configuration? ML2? Other part of neutron? Something else?
>
> Thank you very much
>
>
> Manuel Sopena Ballesteros | Big data Engineer
> Garvan Institute of Medical Research
> The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
> T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb at garvan.org.au
>
> NOTICE
> Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171218/cf18afb1/attachment.html>
More information about the Openstack
mailing list