<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I'm at a loss after quite a bit of testing, so thought I'd see if someone has any ideas…<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We have a Stein cluster that was/is deployed with Kolla Ansible with DVR without OVS+DPDK - rather just the standard kernel OVS.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Everything appears to work well with low-latency local and Internet destinations, but as the latency increases, throughput drops significantly. I would have blamed the standard bandwidth-delay product and TCP window size not being large enough, etc., but that definitely isn't the issue, since we have shown that VMs on our VMware clusters (VLAN-based, not NSX-based) do not have the issue and use the same many-10Gbps-connected ISPs to our routers as the OpenStack configuration.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am using a Vultr machine in London as a test destination since the latency is around 86ms - a good high-latency test.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The good environment we are testing against: Using iperf3 with 1 thread on a VMware VM results in an average of 155Mbits/sec sent to Vultr London with a few retries recorded that lower the window size to around 1.78MiBytes. This is roughly what we expect, if not more, with faster machines.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The problem: Performing the same test on a "much" faster server (E-2288G based, without hyperthreading enabled, so 4.5GHz on all 8 cores), with no other VMs on it, a VM with iperf3 with 1 thread results in an average of only around 32Mbits/sec with a window size that drops significantly at the start and increases to only about 386KiBytes before retries start to happen. This average does not change when using other machines in our cluster, including some Xeon Gold and EPYC processor machines.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The only thing I can point to is OVS and/or something in Linux. However, both tests (VMware and KVM) were performed with the same fresh installation and configuration (CentOS 7 with all updates and the standard kernel). So, my doubts are that it has something to do with Linux.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On the physical host, the vhost process barely registers any CPU usage, and neither does the VM - as expected on such a fast machine.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I even tried to enable multiqueue, since we had it disabled. With a 2 vCPU VM, ethtool -l eth0 displays that 2 queues are in use. Same results - 32Mbits/sec throughput to Vultr London.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>All machines are connected via 100Gbps or 25Gbps connections with Mellanox Connect-X 4 and 5 cards with Mellanox switches, and iperf can saturate these links between the physical hosts when run on the physical hosts' CentOS.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>No errors (tx errors, rx errors, packet loss, etc.) are being reported on any of the logical or physical links.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I thought a queue got full somewhere, so I looked for where queues were used and found that virtio has a tx_queue_size that can be adjusted, but unfortunately, it appears this can only be done with vhost-user, which I believe is specific to OVS+DPDK. The default tx_queue_size in virtio is 256 elements. I'm not sure what this represents, though, much less how to measure the queue usage statistics.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Running iperf3 in "Reverse mode", so the Vultr London machine sends data back to the KVM VM, bandwidth averages around 254Mbits/sec! So the issue appears to be only in one direction - sending network traffic outbound.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Running the same iperf3 test from the KVM VM to a Vultr machine that is very close to us (latency of 2ms or so), results in an average bandwidth of 373Mbits/sec, but with 39 retries per second on average. The window size never adjusts low enough to stop the retries - it averages 146KiBytes in size. Using "Reverse mode" results in an average of 1.21Gbits/sec!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So, the OVS software isn't slow, but something is causing issues when transmitting data when latency is large - something that doesn't happen with the VMware VMs with the same network and Internet connections.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Also, pings from the VMware and KVM VMs are nearly identical - so there isn't some crazy amount of latency caused by OVS - at least when it is essentially idle.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I'm not sure where to go from here in terms of diagnostics. Any ideas?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks for any help, in advance!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Eric<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>