<div dir="ltr"><div><div><div>Hi all,<br><br></div>I have come across a network problem when doing network performance testing. From vm, wget a file on external network is much slower than expected.<br><br><div>The network is setup like this [1].<br>
<br></div><div>To download a file, packets travel from target server public ip to public ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node, via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a long chain.<br>
</div></div><div><br></div><div></div><div>Here are some numbers:<br></div><div>wget <a href="http://mytestserver/big-file.deb">http://mytestserver/big-file.deb</a><br><br></div><div>I use /etc/hosts to set mytestserver to the IP I want to test.<br>
<br></div><div> Download from <a href="http://10.20.165.2">10.20.165.2</a>: 47.2 MB/s (395Mbps) slow but acceptable.<br></div><div> Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps) way too slow.<br>
<br></div><div>I also tried the reverse: run web server on vm, and wget on <a href="http://10.20.165.2">10.20.165.2</a>:<br><br></div><div> Download from vm's floatingip (10.21.144.106) from outside: 105MB/s (880Mbps)<br>
<br></div><div>I have yet to test upload speed to outside network. Upload to <a href="http://10.20.165.0/24">10.20.165.0/24</a> and <a href="http://10.21.144.0/24">10.21.144.0/24</a> are both at 40+MB/s. Upload to public ip not tested yet.<br>
</div><br></div>Here are some raw iperf performance result:<br><br>with "GRO off" on br-ex and eth1:<br>| scenario | outgoing (from vm) | incoming (to vm) |<br>|----------------------------+--------------------+------------------|<br>
| vm to vm on same host | 3.16 Gbps | - |<br>| vm to vm on different host | 613 Mbps | - |<br>| vm to <a href="http://10.21.144.0/24">10.21.144.0/24</a> | 887 Mbps | 860 Mbps |<br>
| vm to <a href="http://10.20.165.0/24">10.20.165.0/24</a> | 888 Mbps | 818 Mbps |<br>| vm to <public ip> | 919 Mbps | 817 Mbps |<br><br>I'm using neutron with ovs plugin and GRE tunnels. There are no errors in neutron's log files, except<br>
<div><br>ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver<br><br></div><div>and similar which has always existed since I deploy openstack.<br><br></div><div>Things I have tried and did not resolve the problem:<br>
<br></div><div> - turn off GRO on neutron node br-ex/eth1 interface (kept off for all wget/scp tests)<br></div><div> - allow all ports in security groups<br></div><div> - drop iptables rules on neutron node<br></div><div>
- lower MTU to 1400 (This helped a little, all wget numbers above have MTU set to 1400 in vm)<br><br></div><div>I also suspected this nat rule is too slow:<br><br> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE<br>
<br></div><div>But test had proved me wrong. I have tested using 10.21.144.5 as the default gateway in a physical server and wget the same file from is lightning fast.<br><br></div><div>So is this download speed typical? If not, how can I isolate the problem and figure out what cause the slowness?<br>
<br></div><div>Thanks,<br>Yuanle<br></div><div><br>[1] <a href="http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png" target="_blank">http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png</a></div></div>