[wallaby][neutron][ovn] Low network performance between instances on different compute nodes using OVN Geneve tunnels
Laurent Dumont
laurentfdumont at gmail.com
Sun Jul 4 16:49:58 UTC 2021
Nothing super specific I can think of but :
- Can you try running the same tests with two instances on the same
compute?
- How many cores are inside the sender/receiver VM?
- Can you test in UDP mode?
On Sun, Jul 4, 2021 at 8:27 AM Malik Obaid <malikobaidadil at gmail.com> wrote:
> Hi,
>
> I am using Openstack Wallaby release with OVN on Ubuntu 20.04.
>
> My environment consists of 2 compute nodes and 1 controller node.
> ovs_version: "2.15.0"
> Ubuntu Kernel Version: 5.4.0-77-generic
>
>
> I am observing Network performance between instances on different compute
> nodes is slow. The network uses geneve tunnels.The environment is using
> 10Gbps network interface cards. However, iperf between instances on
> different compute nodes attains only speeds between a few hundred Mbit/s
> and a few Gb/s. Both instances are in the same tenant network.
>
> Note: iperf results between both compute nodes (hypervisors) across the
> geneve tunnel endpoints is perfect 10 Gbps.
>
> Below are the results of iperf commands.
>
> *iperf server:*
>
> 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc fq_codel state
> UP group default qlen 1000
> link/ether fa:16:3e:4b:1d:29 brd ff:ff:ff:ff:ff:ff
> inet 192.168.100.111/24 brd 192.168.100.255 scope global dynamic ens3
> valid_lft 42694sec preferred_lft 42694sec
> inet6 fe80::f816:3eff:fe4b:1d29/64 scope link
> valid_lft forever preferred_lft forever
>
> root at vm-01:~# iperf3 -s
> Server listening on 5201
>
> Accepted connection from 192.168.100.69, port 45542
> [ 5] local 192.168.100.111 port 5201 connected to 192.168.100.69 port
> 45544
> [ 8] local 192.168.100.111 port 5201 connected to 192.168.100.69 port
> 45546
> [ ID][Role] Interval Transfer Bitrate Retr Cwnd
> [ 5][RX-S] 0.00-1.00 sec 692 MBytes 5.81 Gbits/sec
> [ 8][TX-S] 0.00-1.00 sec 730 MBytes 6.12 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 1.00-2.00 sec 598 MBytes 5.01 Gbits/sec
> [ 8][TX-S] 1.00-2.00 sec 879 MBytes 7.37 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 2.00-3.00 sec 793 MBytes 6.65 Gbits/sec
> [ 8][TX-S] 2.00-3.00 sec 756 MBytes 6.34 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 3.00-4.00 sec 653 MBytes 5.48 Gbits/sec
> [ 8][TX-S] 3.00-4.00 sec 871 MBytes 7.31 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 4.00-5.00 sec 597 MBytes 5.01 Gbits/sec
> [ 8][TX-S] 4.00-5.00 sec 858 MBytes 7.20 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 5.00-6.00 sec 734 MBytes 6.16 Gbits/sec
> [ 8][TX-S] 5.00-6.00 sec 818 MBytes 6.86 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 6.00-7.00 sec 724 MBytes 6.06 Gbits/sec
> [ 8][TX-S] 6.00-7.00 sec 789 MBytes 6.60 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 7.00-8.00 sec 735 MBytes 6.18 Gbits/sec
> [ 8][TX-S] 7.00-8.00 sec 835 MBytes 7.02 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 8.00-9.00 sec 789 MBytes 6.62 Gbits/sec
> [ 8][TX-S] 8.00-9.00 sec 845 MBytes 7.09 Gbits/sec 0 3.14
> MBytes
> [ 5][RX-S] 9.00-10.00 sec 599 MBytes 5.02 Gbits/sec
> [ 8][TX-S] 9.00-10.00 sec 806 MBytes 6.76 Gbits/sec 0 3.14
> MBytes
>
> [ ID][Role] Interval Transfer Bitrate Retr
> [ 5][RX-S] 0.00-10.00 sec 6.75 GBytes 5.80 Gbits/sec
> receiver
> [ 8][TX-S] 0.00-10.00 sec 7.99 GBytes 6.87 Gbits/sec 0 sender
>
> Server listening on 5201
>
> *Client side:*
>
> root at vm-03:~# iperf3 -c 192.168.100.111 --bidir
> Connecting to host 192.168.100.111, port 5201
> [ 5] local 192.168.100.69 port 45544 connected to 192.168.100.111 port
> 5201
> [ 7] local 192.168.100.69 port 45546 connected to 192.168.100.111 port
> 5201
> [ ID][Role] Interval Transfer Bitrate Retr Cwnd
> [ 5][TX-C] 0.00-1.00 sec 700 MBytes 5.87 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 0.00-1.00 sec 722 MBytes 6.06 Gbits/sec
> [ 5][TX-C] 1.00-2.00 sec 594 MBytes 4.98 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 1.00-2.00 sec 883 MBytes 7.41 Gbits/sec
> [ 5][TX-C] 2.00-3.00 sec 796 MBytes 6.67 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 2.00-3.00 sec 752 MBytes 6.31 Gbits/sec
> [ 5][TX-C] 3.00-4.00 sec 654 MBytes 5.49 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 3.00-4.00 sec 876 MBytes 7.35 Gbits/sec
> [ 5][TX-C] 4.00-5.00 sec 598 MBytes 5.01 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 4.00-5.00 sec 853 MBytes 7.16 Gbits/sec
> [ 5][TX-C] 5.00-6.00 sec 734 MBytes 6.15 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 5.00-6.00 sec 818 MBytes 6.86 Gbits/sec
> [ 5][TX-C] 6.00-7.00 sec 726 MBytes 6.09 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 6.00-7.00 sec 793 MBytes 6.65 Gbits/sec
> [ 5][TX-C] 7.00-8.00 sec 734 MBytes 6.15 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 7.00-8.00 sec 831 MBytes 6.97 Gbits/sec
> [ 5][TX-C] 8.00-9.00 sec 788 MBytes 6.61 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 8.00-9.00 sec 845 MBytes 7.09 Gbits/sec
> [ 5][TX-C] 9.00-10.00 sec 600 MBytes 5.03 Gbits/sec 0 3.13
> MBytes
> [ 7][RX-C] 9.00-10.00 sec 805 MBytes 6.76 Gbits/sec
>
> [ ID][Role] Interval Transfer Bitrate Retr
> [ 5][TX-C] 0.00-10.00 sec 6.76 GBytes 5.81 Gbits/sec 0
> sender
> [ 5][TX-C] 0.00-10.00 sec 6.75 GBytes 5.80 Gbits/sec
> receiver
> [ 7][RX-C] 0.00-10.00 sec 7.99 GBytes 6.87 Gbits/sec 0
> sender
> [ 7][RX-C] 0.00-10.00 sec 7.99 GBytes 6.86 Gbits/sec
> receiver
>
> iperf Done.
>
>
> ---------------------------------------------------------------------------------------------------------
>
> *ovs-vsctl show on compute node1:*
>
> root at kvm01-a1-khi01:~# ovs-vsctl show
> 88e6b984-44dc-4f74-8a9a-891742dbbdbd
> Bridge br-eth1
> Port ens224
> Interface ens224
> Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
> Interface
> patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
> type: patch
> options:
> {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}
> Port br-eth1
> Interface br-eth1
> type: internal
> Bridge br-int
> fail_mode: secure
> datapath_type: system
> Port tapde98b2d4-a0
> Interface tapde98b2d4-a0
> Port ovn-f51ef9-0
> Interface ovn-f51ef9-0
> type: vxlan
> options: {csum="true", key=flow, remote_ip="172.16.30.3"}
> bfd_status: {diagnostic="No Diagnostic", flap_count="1",
> forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up,
> state=up}
> Port tap348fc6dc-3a
> Interface tap348fc6dc-3a
> Port br-int
> Interface br-int
> type: internal
> Port tap6d4d8e02-c0
> Interface tap6d4d8e02-c0
> error: "could not open network device tap6d4d8e02-c0 (No
> such device)"
> Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
> Interface
> patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
> type: patch
> options:
> {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}
> Port tap247fe5b2-ff
> Interface tap247fe5b2-ff
>
>
> ------------------------------------------------------------------------------------------------------
>
> *ovs-vsctl show on compute node2:*
>
> root at kvm03-a1-khi01:~# ovs-vsctl show
> 24ce6475-89bb-4df5-a5ff-4ce58f2c2f68
> Bridge br-eth1
> Port patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
> Interface
> patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int
> type: patch
> options:
> {peer=patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9}
> Port br-eth1
> Interface br-eth1
> type: internal
> Port ens224
> Interface ens224
> Bridge br-int
> fail_mode: secure
> datapath_type: system
> Port patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
> Interface
> patch-br-int-to-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9
> type: patch
> options:
> {peer=patch-provnet-440be99a-c347-4458-b7c1-6c0e6155eee9-to-br-int}
> Port tap2b0bbf7b-59
> Interface tap2b0bbf7b-59
> Port ovn-650be8-0
> Interface ovn-650be8-0
> type: vxlan
> options: {csum="true", key=flow, remote_ip="172.16.30.1"}
> bfd_status: {diagnostic="No Diagnostic", flap_count="1",
> forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up,
> state=up}
> Port tap867d2174-83
> Interface tap867d2174-83
> Port tapde98b2d4-a0
> Interface tapde98b2d4-a0
> Port br-int
> Interface br-int
> type: internal
>
>
> --------------------------------------------------------------------------------------------------------
>
> I would really appreciate any input in this regard.
>
> Thank you.
>
> Regards,
> Malik Obaid
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210704/982382cd/attachment-0001.html>
More information about the openstack-discuss
mailing list