<div dir="ltr"><div>Hi Laurent,</div><div><br></div>I am using 32 cores and 32GB RAM on VM. The compute nodes are EPYC 7532 dual socket baremetal servers with 1TB RAM with ubuntu 20.04 and network cards are Broadcom BCM57504 NetXtreme-E 10Gb cards.<br><br>Below are the stats on different hosts.<br><br>TCP bidirectional, on geneve network.<br><br>[ ID][Role] Interval Transfer Bitrate Retr<br>[ 5][RX-S] 0.00-10.01 sec 3.44 GBytes 2.95 Gbits/sec receiver<br>[ 8][TX-S] 0.00-10.01 sec 3.48 GBytes 2.99 Gbits/sec 0 sender<br><br>Unidirectional tcp, on geneve network.<br><br>[ ID] Interval Transfer Bitrate Retr<br>[ 5] 0.00-10.00 sec 6.29 GBytes 5.41 Gbits/sec 491 sender<br>[ 5] 0.00-10.00 sec 6.29 GBytes 5.40 Gbits/sec receiver<br><br>Unidirectional udp, on geneve network.<br><br>[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams<br>[ 5] 0.00-10.00 sec 3.45 GBytes 2.97 Gbits/sec 0.000 ms 0/2540389 (0%) sender<br>[ 5] 0.00-10.00 sec 3.23 GBytes 2.77 Gbits/sec 0.009 ms 165868/2539198 (6.5%) receiver<br><br>Below are the stats of bidirectional udp, on geneve network.<br><br>[ ID][Role] Interval Transfer Bitrate Jitter Lost/Total Datagrams<br>[ 5][TX-C] 0.00-10.00 sec 2.00 GBytes 1.72 Gbits/sec 0.000 ms 0/1472357 (0%) sender<br>[ 5][TX-C] 0.00-10.01 sec 1.99 GBytes 1.71 Gbits/sec 0.024 ms 7713/1471535 (0.52%) receiver<br>[ 7][RX-C] 0.00-10.00 sec 2.00 GBytes 1.72 Gbits/sec 0.000 ms 0/1472450 (0%) sender<br>[ 7][RX-C] 0.00-10.01 sec 1.98 GBytes 1.70 Gbits/sec 0.012 ms 17325/1470552 (1.2%) receiver<br><br>==================================================<br><br>Below are the stats of VMs on same host.<br><br>tcp unidirectional on geneve network.<br><br>[ ID] Interval Transfer Bitrate Retr<br>[ 5] 0.00-10.00 sec 19.5 GBytes 16.7 Gbits/sec 0 sender<br>[ 5] 0.00-10.00 sec 19.5 GBytes 16.7 Gbits/sec receiver<br><br>tcp bidirectional on geneve network.<br><br>[ ID][Role] Interval Transfer Bitrate Retr<br>[ 5][TX-C] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec 0 sender<br>[ 5][TX-C] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec receiver<br>[ 7][RX-C] 0.00-10.00 sec 9.95 GBytes 8.55 Gbits/sec 0 sender<br>[ 7][RX-C] 0.00-10.00 sec 9.95 GBytes 8.54 Gbits/sec receiver<br><br>udp unidirectional on geneve network.<br><br>[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams<br>[ 5] 0.00-10.00 sec 2.15 GBytes 1.85 Gbits/sec 0.000 ms 0/1584825 (0%) sender<br>[ 5] 0.00-10.00 sec 2.15 GBytes 1.85 Gbits/sec 0.015 ms 0/1584825 (0%) receiver<br><br>udp bidirectional on geneve network.<br><br>[ ID][Role] Interval Transfer Bitrate Jitter Lost/Total Datagrams<br>[ 5][TX-C] 0.00-10.00 sec 2.17 GBytes 1.87 Gbits/sec 0.000 ms 0/1597563 (0%) sender<br>[ 5][TX-C] 0.00-10.00 sec 1.37 GBytes 1.17 Gbits/sec 0.006 ms 590524/1595459 (37%) receiver<br>[ 7][RX-C] 0.00-10.00 sec 1.37 GBytes 1.17 Gbits/sec 0.000 ms 0/1005024 (0%) sender<br>[ 7][RX-C] 0.00-10.00 sec 1.37 GBytes 1.17 Gbits/sec 0.012 ms 0/1004983 (0%) receiver<br><br>However the performance of network on OVN VLAN provider network is ~9.8Gbps bidirectional with VMs on different hosts.<br><br>Below are the details of ovs-vsctl show command on compute node.<br><br> Bridge br-int<br> fail_mode: secure<br> datapath_type: system<br> Port tap131797c7-06<br> Interface tap131797c7-06<br> Port ovn-094381-0<br> Interface ovn-094381-0<br> type: geneve<br> options: {csum="true", key=flow, remote_ip="172.16.40.2"}<br> bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}<br> Port patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d<br> Interface patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d<br> type: patch<br> options: {peer=patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int}<br> Port br-int<br> Interface br-int<br> type: internal<br> Port tap18ca5a79-10<br> Interface tap18ca5a79-10<br> Bridge br-vlan<br> Port bond0<br> Interface bond0<br> Port patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int<br> Interface patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int<br> type: patch<br> options: {peer=patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d}<br> Port br-vlan<br> Interface br-vlan<br> type: internal<br> ovs_version: "2.15.0"<br><br>I have already tuned the BIOS for max performance. Any tuning required at OVS or OS level. I strongly believe that throughput should be 10Gbps without dpdk on geneve network.<br><div><br></div><div>Regards,</div><div>Malik Obaid</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 4, 2021 at 9:50 PM Laurent Dumont <<a href="mailto:laurentfdumont@gmail.com">laurentfdumont@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Nothing super specific I can think of but :<div><ul><li>Can you try running the same tests with two instances on the same compute?</li><li>How many cores are inside the sender/receiver VM?</li><li>Can you test in UDP mode?</li></ul><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 4, 2021 at 8:27 AM Malik Obaid <<a href="mailto:malikobaidadil@gmail.com" target="_blank">malikobaidadil@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><br><div>I am using Openstack Wallaby release with OVN on Ubuntu 20.04.</div><div><br></div><div>My environment consists of 2 compute nodes and 1 controller node.</div><div>ovs_version: "2.15.0"</div><div>Ubuntu Kernel Version: 5.4.0-77-generic<br></div><div><br></div><div><br></div><div>I am observing Network performance between instances on different compute nodes is slow. The network uses geneve tunnels.The environment is using 10Gbps network interface cards. However, iperf between instances on different compute nodes attains only speeds between a few hundred Mbit/s and a few Gb/s. Both instances are in the same tenant network.</div><div>
<p>Note: iperf results between both compute nodes (hypervisors) across the geneve tunnel endpoints is perfect 10 Gbps.</p><p>Below are the results of iperf commands. <br></p><p><font size="4"><b>iperf server:</b></font><br><br>2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UP group default qlen 1000<br> link/ether fa:16:3e:4b:1d:29 brd ff:ff:ff:ff:ff:ff<br> inet <a href="http://192.168.100.111/24" target="_blank">192.168.100.111/24</a> brd 192.168.100.255 scope global dynamic ens3<br> valid_lft 42694sec preferred_lft 42694sec<br> inet6 fe80::f816:3eff:fe4b:1d29/64 scope link<br> valid_lft forever preferred_lft forever</p><p>root@vm-01:~# iperf3 -s<br>Server listening on 5201<br><br>Accepted connection from 192.168.100.69, port 45542<br>[ 5] local 192.168.100.111 port 5201 connected to 192.168.100.69 port 45544<br>[ 8] local 192.168.100.111 port 5201 connected to 192.168.100.69 port 45546<br>[ ID][Role] Interval Transfer Bitrate Retr Cwnd<br>[ 5][RX-S] 0.00-1.00 sec 692 MBytes 5.81 Gbits/sec<br>[ 8][TX-S] 0.00-1.00 sec 730 MBytes 6.12 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 1.00-2.00 sec 598 MBytes 5.01 Gbits/sec<br>[ 8][TX-S] 1.00-2.00 sec 879 MBytes 7.37 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 2.00-3.00 sec 793 MBytes 6.65 Gbits/sec<br>[ 8][TX-S] 2.00-3.00 sec 756 MBytes 6.34 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 3.00-4.00 sec 653 MBytes 5.48 Gbits/sec<br>[ 8][TX-S] 3.00-4.00 sec 871 MBytes 7.31 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 4.00-5.00 sec 597 MBytes 5.01 Gbits/sec<br>[ 8][TX-S] 4.00-5.00 sec 858 MBytes 7.20 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 5.00-6.00 sec 734 MBytes 6.16 Gbits/sec<br>[ 8][TX-S] 5.00-6.00 sec 818 MBytes 6.86 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 6.00-7.00 sec 724 MBytes 6.06 Gbits/sec<br>[ 8][TX-S] 6.00-7.00 sec 789 MBytes 6.60 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 7.00-8.00 sec 735 MBytes 6.18 Gbits/sec<br>[ 8][TX-S] 7.00-8.00 sec 835 MBytes 7.02 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 8.00-9.00 sec 789 MBytes 6.62 Gbits/sec<br>[ 8][TX-S] 8.00-9.00 sec 845 MBytes 7.09 Gbits/sec 0 3.14 MBytes<br>[ 5][RX-S] 9.00-10.00 sec 599 MBytes 5.02 Gbits/sec<br>[ 8][TX-S] 9.00-10.00 sec 806 MBytes 6.76 Gbits/sec 0 3.14 MBytes<br><br>[ ID][Role] Interval Transfer Bitrate Retr<br>[ 5][RX-S] 0.00-10.00 sec 6.75 GBytes 5.80 Gbits/sec receiver<br>[ 8][TX-S] 0.00-10.00 sec 7.99 GBytes 6.87 Gbits/sec 0 sender<br><br>Server listening on 5201</p><p><font size="4"><b>Client side:</b></font><br><br>root@vm-03:~# iperf3 -c 192.168.100.111 --bidir<br>Connecting to host 192.168.100.111, port 5201<br>[ 5] local 192.168.100.69 port 45544 connected to 192.168.100.111 port 5201<br>[ 7] local 192.168.100.69 port 45546 connected to 192.168.100.111 port 5201<br>[ ID][Role] Interval Transfer Bitrate Retr Cwnd<br>[ 5][TX-C] 0.00-1.00 sec 700 MBytes 5.87 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 0.00-1.00 sec 722 MBytes 6.06 Gbits/sec<br>[ 5][TX-C] 1.00-2.00 sec 594 MBytes 4.98 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 1.00-2.00 sec 883 MBytes 7.41 Gbits/sec<br>[ 5][TX-C] 2.00-3.00 sec 796 MBytes 6.67 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 2.00-3.00 sec 752 MBytes 6.31 Gbits/sec<br>[ 5][TX-C] 3.00-4.00 sec 654 MBytes 5.49 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 3.00-4.00 sec 876 MBytes 7.35 Gbits/sec<br>[ 5][TX-C] 4.00-5.00 sec 598 MBytes 5.01 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 4.00-5.00 sec 853 MBytes 7.16 Gbits/sec<br>[ 5][TX-C] 5.00-6.00 sec 734 MBytes 6.15 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 5.00-6.00 sec 818 MBytes 6.86 Gbits/sec<br>[ 5][TX-C] 6.00-7.00 sec 726 MBytes 6.09 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 6.00-7.00 sec 793 MBytes 6.65 Gbits/sec<br>[ 5][TX-C] 7.00-8.00 sec 734 MBytes 6.15 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 7.00-8.00 sec 831 MBytes 6.97 Gbits/sec<br>[ 5][TX-C] 8.00-9.00 sec 788 MBytes 6.61 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 8.00-9.00 sec 845 MBytes 7.09 Gbits/sec<br>[ 5][TX-C] 9.00-10.00 sec 600 MBytes 5.03 Gbits/sec 0 3.13 MBytes<br>[ 7][RX-C] 9.00-10.00 sec 805 MBytes 6.76 Gbits/sec<br><br>[ ID][Role] Interval Transfer Bitrate Retr<br>[ 5][TX-C] 0.00-10.00 sec 6.76 GBytes 5.81 Gbits/sec 0 sender<br>[ 5][TX-C] 0.00-10.00 sec 6.75 GBytes 5.80 Gbits/sec receiver<br>[ 7][RX-C] 0.00-10.00 sec 7.99 GBytes 6.87 Gbits/sec 0 sender<br>[ 7][RX-C] 0.00-10.00 sec 7.99 GBytes 6.86 Gbits/sec receiver<br><br>iperf Done.</p><p>---------------------------------------------------------------------------------------------------------<br></p><p><font size="4"><b>ovs-vsctl show on compute node1:</b></font><br></p><p>root@kvm01-a1-khi01:~# ovs-vsctl show<br>88e6b984-44dc-4f74-8a9a-891742dbbdbd<br> Bridge br-eth1<br> Port ens224<br> Interface ens224<br> Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int<br> Interface patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int<br> type: patch<br> options: {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}<br> Port br-eth1<br> Interface br-eth1<br> type: internal<br> Bridge br-int<br> fail_mode: secure<br> datapath_type: system<br> Port tapde98b2d4-a0<br> Interface tapde98b2d4-a0<br> Port ovn-f51ef9-0<br> Interface ovn-f51ef9-0<br> type: vxlan<br> options: {csum="true", key=flow, remote_ip="172.16.30.3"}<br> bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}<br> Port tap348fc6dc-3a<br> Interface tap348fc6dc-3a<br> Port br-int<br> Interface br-int<br> type: internal<br> Port tap6d4d8e02-c0<br> Interface tap6d4d8e02-c0<br> error: "could not open network device tap6d4d8e02-c0 (No such device)"<br> Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9<br> Interface patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9<br> type: patch<br> options: {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}<br> Port tap247fe5b2-ff<br> Interface tap247fe5b2-ff</p><p>------------------------------------------------------------------------------------------------------<br></p><p><font size="4"><b>ovs-vsctl show on compute node2:</b></font></p><p>root@kvm03-a1-khi01:~# ovs-vsctl show<br>24ce6475-89bb-4df5-a5ff-4ce58f2c2f68<br> Bridge br-eth1<br> Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int<br> Interface patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int<br> type: patch<br> options: {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}<br> Port br-eth1<br> Interface br-eth1<br> type: internal<br> Port ens224<br> Interface ens224<br> Bridge br-int<br> fail_mode: secure<br> datapath_type: system<br> Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9<br> Interface patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9<br> type: patch<br> options: {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}<br> Port tap2b0bbf7b-59<br> Interface tap2b0bbf7b-59<br> Port ovn-650be8-0<br> Interface ovn-650be8-0<br> type: vxlan<br> options: {csum="true", key=flow, remote_ip="172.16.30.1"}<br> bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}<br> Port tap867d2174-83<br> Interface tap867d2174-83<br> Port tapde98b2d4-a0<br> Interface tapde98b2d4-a0<br> Port br-int<br> Interface br-int<br> type: internal</p><p>--------------------------------------------------------------------------------------------------------</p><p>I would really appreciate any input in this regard.<br><br>Thank you.<br><br>Regards,<br>Malik Obaid</p><p><br></p></div></div>
</blockquote></div>
</blockquote></div>