I think that you should be able to reach line rate as well for 10Gbit on Geneve/OVN. I don't have a setup to compare, but you might want to try to force the TCP Window with iperf3. 

There could be a case where the PPS (packet per second) is the issue and it cannot reach a sufficiently big window.

On Mon, Jul 5, 2021 at 2:19 AM Malik Obaid <malikobaidadil@gmail.com> wrote:
Hi Laurent,

I am using 32 cores and 32GB RAM on VM. The compute nodes are EPYC 7532 dual socket baremetal servers with 1TB RAM with ubuntu 20.04 and network cards are Broadcom BCM57504 NetXtreme-E 10Gb cards.

Below are the stats on different hosts.

TCP bidirectional, on geneve network.

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][RX-S]   0.00-10.01  sec  3.44 GBytes  2.95 Gbits/sec                  receiver
[  8][TX-S]   0.00-10.01  sec  3.48 GBytes  2.99 Gbits/sec    0             sender

Unidirectional tcp, on geneve network.

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.29 GBytes  5.41 Gbits/sec  491             sender
[  5]   0.00-10.00  sec  6.29 GBytes  5.40 Gbits/sec                  receiver

Unidirectional udp, on geneve network.

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  3.45 GBytes  2.97 Gbits/sec  0.000 ms  0/2540389 (0%)  sender
[  5]   0.00-10.00  sec  3.23 GBytes  2.77 Gbits/sec  0.009 ms  165868/2539198 (6.5%)  receiver

Below are the stats of bidirectional udp, on geneve network.

[ ID][Role] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5][TX-C]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec  0.000 ms  0/1472357 (0%)  sender
[  5][TX-C]   0.00-10.01  sec  1.99 GBytes  1.71 Gbits/sec  0.024 ms  7713/1471535 (0.52%)  receiver
[  7][RX-C]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec  0.000 ms  0/1472450 (0%)  sender
[  7][RX-C]   0.00-10.01  sec  1.98 GBytes  1.70 Gbits/sec  0.012 ms  17325/1470552 (1.2%)  receiver

==================================================

Below are the stats of VMs on same host.

tcp unidirectional on geneve network.

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  19.5 GBytes  16.7 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  19.5 GBytes  16.7 Gbits/sec                  receiver

tcp bidirectional on geneve network.

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  10.7 GBytes  9.21 Gbits/sec    0             sender
[  5][TX-C]   0.00-10.00  sec  10.7 GBytes  9.21 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  9.95 GBytes  8.55 Gbits/sec    0             sender
[  7][RX-C]   0.00-10.00  sec  9.95 GBytes  8.54 Gbits/sec                  receiver

udp unidirectional on geneve network.

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.15 GBytes  1.85 Gbits/sec  0.000 ms  0/1584825 (0%)  sender
[  5]   0.00-10.00  sec  2.15 GBytes  1.85 Gbits/sec  0.015 ms  0/1584825 (0%)  receiver

udp bidirectional on geneve network.

[ ID][Role] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5][TX-C]   0.00-10.00  sec  2.17 GBytes  1.87 Gbits/sec  0.000 ms  0/1597563 (0%)  sender
[  5][TX-C]   0.00-10.00  sec  1.37 GBytes  1.17 Gbits/sec  0.006 ms  590524/1595459 (37%)  receiver
[  7][RX-C]   0.00-10.00  sec  1.37 GBytes  1.17 Gbits/sec  0.000 ms  0/1005024 (0%)  sender
[  7][RX-C]   0.00-10.00  sec  1.37 GBytes  1.17 Gbits/sec  0.012 ms  0/1004983 (0%)  receiver

However the performance of network on OVN VLAN provider network is ~9.8Gbps bidirectional with VMs on different hosts.

Below are the details of ovs-vsctl show command on compute node.

    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port tap131797c7-06
            Interface tap131797c7-06
        Port ovn-094381-0
            Interface ovn-094381-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.16.40.2"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d
            Interface patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d
                type: patch
                options: {peer=patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int}
        Port br-int
            Interface br-int
                type: internal
        Port tap18ca5a79-10
            Interface tap18ca5a79-10
    Bridge br-vlan
        Port bond0
            Interface bond0
        Port patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int
            Interface patch-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-4f09b820-b5c1-4006-bb6a-a14ecf7f776d}
        Port br-vlan
            Interface br-vlan
                type: internal
    ovs_version: "2.15.0"

I have already tuned the BIOS for max performance. Any tuning required at OVS or OS level. I strongly believe that throughput should be 10Gbps without dpdk on geneve network.

Regards,
Malik Obaid

On Sun, Jul 4, 2021 at 9:50 PM Laurent Dumont <laurentfdumont@gmail.com> wrote:
Nothing super specific I can think of but :
  • Can you try running the same tests with two instances on the same compute?
  • How many cores are inside the sender/receiver VM?
  • Can you test in UDP mode?


On Sun, Jul 4, 2021 at 8:27 AM Malik Obaid <malikobaidadil@gmail.com> wrote:
Hi,

I am using Openstack Wallaby release with OVN on Ubuntu 20.04.

My environment consists of 2 compute nodes and 1 controller node.
ovs_version: "2.15.0"
Ubuntu Kernel Version: 5.4.0-77-generic


I am observing Network performance between instances on different compute nodes is slow. The network uses geneve tunnels.The environment is using 10Gbps network interface cards. However, iperf between instances on different compute nodes attains only speeds between a few hundred Mbit/s and a few Gb/s. Both instances are in the same tenant network.

Note: iperf results between both compute nodes (hypervisors) across the geneve tunnel endpoints is perfect 10 Gbps.

Below are the results of iperf commands.

iperf server:

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:4b:1d:29 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.111/24 brd 192.168.100.255 scope global dynamic ens3
       valid_lft 42694sec preferred_lft 42694sec
    inet6 fe80::f816:3eff:fe4b:1d29/64 scope link
       valid_lft forever preferred_lft forever

root@vm-01:~# iperf3 -s
Server listening on 5201

Accepted connection from 192.168.100.69, port 45542
[  5] local 192.168.100.111 port 5201 connected to 192.168.100.69 port 45544
[  8] local 192.168.100.111 port 5201 connected to 192.168.100.69 port 45546
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][RX-S]   0.00-1.00   sec   692 MBytes  5.81 Gbits/sec
[  8][TX-S]   0.00-1.00   sec   730 MBytes  6.12 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   1.00-2.00   sec   598 MBytes  5.01 Gbits/sec
[  8][TX-S]   1.00-2.00   sec   879 MBytes  7.37 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   2.00-3.00   sec   793 MBytes  6.65 Gbits/sec
[  8][TX-S]   2.00-3.00   sec   756 MBytes  6.34 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   3.00-4.00   sec   653 MBytes  5.48 Gbits/sec
[  8][TX-S]   3.00-4.00   sec   871 MBytes  7.31 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   4.00-5.00   sec   597 MBytes  5.01 Gbits/sec
[  8][TX-S]   4.00-5.00   sec   858 MBytes  7.20 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   5.00-6.00   sec   734 MBytes  6.16 Gbits/sec
[  8][TX-S]   5.00-6.00   sec   818 MBytes  6.86 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   6.00-7.00   sec   724 MBytes  6.06 Gbits/sec
[  8][TX-S]   6.00-7.00   sec   789 MBytes  6.60 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   7.00-8.00   sec   735 MBytes  6.18 Gbits/sec
[  8][TX-S]   7.00-8.00   sec   835 MBytes  7.02 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   8.00-9.00   sec   789 MBytes  6.62 Gbits/sec
[  8][TX-S]   8.00-9.00   sec   845 MBytes  7.09 Gbits/sec    0   3.14 MBytes
[  5][RX-S]   9.00-10.00  sec   599 MBytes  5.02 Gbits/sec
[  8][TX-S]   9.00-10.00  sec   806 MBytes  6.76 Gbits/sec    0   3.14 MBytes

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][RX-S]   0.00-10.00  sec  6.75 GBytes  5.80 Gbits/sec                  receiver
[ 8][TX-S] 0.00-10.00 sec 7.99 GBytes 6.87 Gbits/sec 0 sender

Server listening on 5201

Client side:

root@vm-03:~# iperf3 -c 192.168.100.111 --bidir
Connecting to host 192.168.100.111, port 5201
[  5] local 192.168.100.69 port 45544 connected to 192.168.100.111 port 5201
[  7] local 192.168.100.69 port 45546 connected to 192.168.100.111 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   700 MBytes  5.87 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   0.00-1.00   sec   722 MBytes  6.06 Gbits/sec
[  5][TX-C]   1.00-2.00   sec   594 MBytes  4.98 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   1.00-2.00   sec   883 MBytes  7.41 Gbits/sec
[  5][TX-C]   2.00-3.00   sec   796 MBytes  6.67 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   2.00-3.00   sec   752 MBytes  6.31 Gbits/sec
[  5][TX-C]   3.00-4.00   sec   654 MBytes  5.49 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   3.00-4.00   sec   876 MBytes  7.35 Gbits/sec
[  5][TX-C]   4.00-5.00   sec   598 MBytes  5.01 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   4.00-5.00   sec   853 MBytes  7.16 Gbits/sec
[  5][TX-C]   5.00-6.00   sec   734 MBytes  6.15 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   5.00-6.00   sec   818 MBytes  6.86 Gbits/sec
[  5][TX-C]   6.00-7.00   sec   726 MBytes  6.09 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   6.00-7.00   sec   793 MBytes  6.65 Gbits/sec
[  5][TX-C]   7.00-8.00   sec   734 MBytes  6.15 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   7.00-8.00   sec   831 MBytes  6.97 Gbits/sec
[  5][TX-C]   8.00-9.00   sec   788 MBytes  6.61 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   8.00-9.00   sec   845 MBytes  7.09 Gbits/sec
[  5][TX-C]   9.00-10.00  sec   600 MBytes  5.03 Gbits/sec    0   3.13 MBytes
[  7][RX-C]   9.00-10.00  sec   805 MBytes  6.76 Gbits/sec

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  6.76 GBytes  5.81 Gbits/sec    0             sender
[  5][TX-C]   0.00-10.00  sec  6.75 GBytes  5.80 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  7.99 GBytes  6.87 Gbits/sec    0             sender
[  7][RX-C]   0.00-10.00  sec  7.99 GBytes  6.86 Gbits/sec                  receiver

iperf Done.

---------------------------------------------------------------------------------------------------------

ovs-vsctl show on compute node1:

root@kvm01-a1-khi01:~# ovs-vsctl show
88e6b984-44dc-4f74-8a9a-891742dbbdbd
    Bridge br-eth1
        Port ens224
            Interface ens224
        Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
            Interface patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}
        Port br-eth1
            Interface br-eth1
                type: internal
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port tapde98b2d4-a0
            Interface tapde98b2d4-a0
        Port ovn-f51ef9-0
            Interface ovn-f51ef9-0
                type: vxlan
                options: {csum="true", key=flow, remote_ip="172.16.30.3"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port tap348fc6dc-3a
            Interface tap348fc6dc-3a
        Port br-int
            Interface br-int
                type: internal
        Port tap6d4d8e02-c0
            Interface tap6d4d8e02-c0
                error: "could not open network device tap6d4d8e02-c0 (No such device)"
        Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
            Interface patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
                type: patch
                options: {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}
        Port tap247fe5b2-ff
            Interface tap247fe5b2-ff

------------------------------------------------------------------------------------------------------

ovs-vsctl show on compute node2:

root@kvm03-a1-khi01:~# ovs-vsctl show
24ce6475-89bb-4df5-a5ff-4ce58f2c2f68
    Bridge br-eth1
        Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
            Interface patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}
        Port br-eth1
            Interface br-eth1
                type: internal
        Port ens224
            Interface ens224
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
            Interface patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
                type: patch
                options: {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}
        Port tap2b0bbf7b-59
            Interface tap2b0bbf7b-59
        Port ovn-650be8-0
            Interface ovn-650be8-0
                type: vxlan
                options: {csum="true", key=flow, remote_ip="172.16.30.1"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port tap867d2174-83
            Interface tap867d2174-83
        Port tapde98b2d4-a0
            Interface tapde98b2d4-a0
        Port br-int
            Interface br-int
                type: internal

--------------------------------------------------------------------------------------------------------

I would really appreciate any input in this regard.

Thank you.

Regards,
Malik Obaid