<div dir="ltr">Hi!<div><br></div><div>Maybe you're facing this problem:</div><div><br></div><div><a href="https://bugs.launchpad.net/neutron/+bug/1252900">https://bugs.launchpad.net/neutron/+bug/1252900</a><br></div><div>
<br></div><div>I'm seeing that you tried to turn GRO off but, is your Network Node a Virtual Machine, or a physical server? If it is a VM (KVM), you'll need to turn GRO off at the hypervisor too.</div><div><br></div>
<div>Happy new year!</div><div><br></div><div>Cheers!</div><div>Thiago</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 31 December 2013 13:55, sylecn <span dir="ltr"><<a href="mailto:sylecn@gmail.com" target="_blank">sylecn@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi all,<br><br></div>I have come across a network problem when doing network performance testing. From vm, wget a file on external network is much slower than expected.<br>
<br><div>The network is setup like this [1].<br>
<br></div><div>To download a file, packets travel from target server public ip to public ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node, via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a long chain.<br>
</div></div><div><br></div><div></div><div>Here are some numbers:<br></div><div>wget <a href="http://mytestserver/big-file.deb" target="_blank">http://mytestserver/big-file.deb</a><br><br></div><div>I use /etc/hosts to set mytestserver to the IP I want to test.<br>
<br></div><div> Download from <a href="http://10.20.165.2" target="_blank">10.20.165.2</a>: 47.2 MB/s (395Mbps) slow but acceptable.<br></div><div> Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps) way too slow.<br>
<br></div><div>I also tried the reverse: run web server on vm, and wget on <a href="http://10.20.165.2" target="_blank">10.20.165.2</a>:<br><br></div><div> Download from vm's floatingip (10.21.144.106) from outside: 105MB/s (880Mbps)<br>
<br></div><div>I have yet to test upload speed to outside network. Upload to <a href="http://10.20.165.0/24" target="_blank">10.20.165.0/24</a> and <a href="http://10.21.144.0/24" target="_blank">10.21.144.0/24</a> are both at 40+MB/s. Upload to public ip not tested yet.<br>
</div><br></div>Here are some raw iperf performance result:<br><br>with "GRO off" on br-ex and eth1:<br>| scenario | outgoing (from vm) | incoming (to vm) |<br>|----------------------------+--------------------+------------------|<br>
| vm to vm on same host | 3.16 Gbps | - |<br>| vm to vm on different host | 613 Mbps | - |<br>| vm to <a href="http://10.21.144.0/24" target="_blank">10.21.144.0/24</a> | 887 Mbps | 860 Mbps |<br>
| vm to <a href="http://10.20.165.0/24" target="_blank">10.20.165.0/24</a> | 888 Mbps | 818 Mbps |<br>| vm to <public ip> | 919 Mbps | 817 Mbps |<br><br>I'm using neutron with ovs plugin and GRE tunnels. There are no errors in neutron's log files, except<br>
<div><br>ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver<br><br></div><div>and similar which has always existed since I deploy openstack.<br><br></div><div>Things I have tried and did not resolve the problem:<br>
<br></div><div> - turn off GRO on neutron node br-ex/eth1 interface (kept off for all wget/scp tests)<br></div><div> - allow all ports in security groups<br></div><div> - drop iptables rules on neutron node<br></div><div>
- lower MTU to 1400 (This helped a little, all wget numbers above have MTU set to 1400 in vm)<br><br></div><div>I also suspected this nat rule is too slow:<br><br> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE<br>
<br></div><div>But test had proved me wrong. I have tested using 10.21.144.5 as the default gateway in a physical server and wget the same file from is lightning fast.<br><br></div><div>So is this download speed typical? If not, how can I isolate the problem and figure out what cause the slowness?<br>
<br></div><div>Thanks,<br>Yuanle<br></div><div><br>[1] <a href="http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png" target="_blank">http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png</a></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>