[ops] Bandwidth problem on computes
Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming. From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this : compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller. compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes. I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance, Jahson
Hi, can you repeat your tests with * iperf from compute1 -> compute2 * iperf from compute2 -> compute1 * ip -r output of both nodes * watching top while doing the iperf and reporting the process using most cpu? * provding ethtool -k <nic> for all nics in compute1+2 Fabian Am Di., 16. März 2021 um 14:49 Uhr schrieb Jahson Babel <jahson.babel@cc.in2p3.fr>:
Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming.
From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this :
compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes
Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller.
compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes
If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes.
I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs
Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance,
Jahson
Hi Fabian, Thank you for taking the time to respond. Here are all the things you asked : - compute1 => compute2 iperf, route, top : https://pastebin.com/fZ54xx19 - compute2 => compute1 iperf, route, top : https://pastebin.com/EZ2FZPCq - compute1 nic ethtool : https://pastebin.com/MVVFVuDj - compute2 nic ethtool : https://pastebin.com/Tb9zQVaf For the ethtool part consider picking only compute1 because I've already tried to play a little bit with GRO/GSO on compute2 without seeing any improvements so far. That's why it is different. The configuration on the compute1 is the default on our hypervisors. Plus I didn't make the ethtool on the VMs's interfaces. I picked the teaming interface, the management and the tunnel interface. In my opinion the tunnel interfaces should not matter but I included anyway. Can the high load from ksoftirqd lead to a such impact on bandwidth ? Let me know if you need something else. Jahson On 19/03/2021 11:45, Fabian Zimmermann wrote:
Hi,
can you repeat your tests with
* iperf from compute1 -> compute2 * iperf from compute2 -> compute1 * ip -r output of both nodes * watching top while doing the iperf and reporting the process using most cpu? * provding ethtool -k <nic> for all nics in compute1+2
Fabian
Am Di., 16. März 2021 um 14:49 Uhr schrieb Jahson Babel <jahson.babel@cc.in2p3.fr>:
Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming.
From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this :
compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes
Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller.
compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes
If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes.
I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs
Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance,
Jahson
participants (2)
-
Fabian Zimmermann
-
Jahson Babel