Open Stack

Hi,

 

I'm at a loss after quite a bit of testing, so thought I'd see if
someone has any ideas...

 

We have a Stein cluster that was/is deployed with Kolla Ansible with DVR
without OVS+DPDK - rather just the standard kernel OVS.

 

Everything appears to work well with low-latency local and Internet
destinations, but as the latency increases, throughput drops
significantly.  I would have blamed the standard bandwidth-delay product
and TCP window size not being large enough, etc., but that definitely
isn't the issue, since we have shown that VMs on our VMware clusters
(VLAN-based, not NSX-based) do not have the issue and use the same
many-10Gbps-connected ISPs to our routers as the OpenStack
configuration.

 

I am using a Vultr machine in London as a test destination since the
latency is around 86ms - a good high-latency test.

 

The good environment we are testing against:  Using iperf3 with 1 thread
on a VMware VM results in an average of 155Mbits/sec sent to Vultr
London with a few retries recorded that lower the window size to around
1.78MiBytes.  This is roughly what we expect, if not more, with faster
machines.

 

The problem:  Performing the same test on a "much" faster server
(E-2288G based, without hyperthreading enabled, so 4.5GHz on all 8
cores), with no other VMs on it, a VM with iperf3 with 1 thread results
in an average of only around 32Mbits/sec with a window size that drops
significantly at the start and increases to only about 386KiBytes before
retries start to happen.  This average does not change when using other
machines in our cluster, including some Xeon Gold and EPYC processor
machines.

 

The only thing I can point to is OVS and/or something in Linux.
However, both tests (VMware and KVM) were performed with the same fresh
installation and configuration (CentOS 7 with all updates and the
standard kernel).  So, my doubts are that it has something to do with
Linux.

 

On the physical host, the vhost process barely registers any CPU usage,
and neither does the VM - as expected on such a fast machine.

 

I even tried to enable multiqueue, since we had it disabled.  With a 2
vCPU VM, ethtool -l eth0 displays that 2 queues are in use.  Same
results - 32Mbits/sec throughput to Vultr London.

 

All machines are connected via 100Gbps or 25Gbps connections with
Mellanox Connect-X 4 and 5 cards with Mellanox switches, and iperf can
saturate these links between the physical hosts when run on the physical
hosts' CentOS.

 

No errors (tx errors, rx errors, packet loss, etc.) are being reported
on any of the logical or physical links.

 

I thought a queue got full somewhere, so I looked for where queues were
used and found that virtio has a tx_queue_size that can be adjusted, but
unfortunately, it appears this can only be done with vhost-user, which I
believe is specific to OVS+DPDK.  The default tx_queue_size in virtio is
256 elements.  I'm not sure what this represents, though, much less how
to measure the queue usage statistics.

 

Running iperf3 in "Reverse mode", so the Vultr London machine sends data
back to the KVM VM, bandwidth averages around 254Mbits/sec!  So the
issue appears to be only in one direction - sending network traffic
outbound.

 

Running the same iperf3 test from the KVM VM to a Vultr machine that is
very close to us (latency of 2ms or so), results in an average bandwidth
of 373Mbits/sec, but with 39 retries per second on average.  The window
size never adjusts low enough to stop the retries - it averages
146KiBytes in size.  Using "Reverse mode" results in an average of
1.21Gbits/sec!

 

So, the OVS software isn't slow, but something is causing issues when
transmitting data when latency is large - something that doesn't happen
with the VMware VMs with the same network and Internet connections.

 

Also, pings from the VMware and KVM VMs are nearly identical - so there
isn't some crazy amount of latency caused by OVS - at least when it is
essentially idle.

 

I'm not sure where to go from here in terms of diagnostics.  Any ideas?

 

Thanks for any help, in advance!

 

Eric

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200428/e1496edf/attachment.html>