<div dir="ltr"><div><div>Hi all,<br><br>I have found the cause of the problem. When you use eth1 as br-ex where eth1 is not connected to internet, and use an iptables NAT rule as software gateway to internet, you need to turn off GRO on both eth0 and eth1.<br>
<br></div>After turning off GRO on eth0, download speed inside vm has become normal.<br><br></div><div>Thanks,<br></div>Yuanle<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 1, 2014 at 1:29 PM, sylecn <span dir="ltr"><<a href="mailto:sylecn@gmail.com" target="_blank">sylecn@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><br></div>Hi Thiago,<br><br></div>The Network Node is a physical server. GRO does affect network much. When GRO is on on the neutron node, from outside to VM bandwidth is about 60Mbps. When it is turned off, from outside to VM bandwidth is over 800Mbps. This is tested using iperf, physical network is 1Gbps.<br>
<br></div><div>Happy new year to you too!<br></div><div><br></div>Thanks,<br>Yuanle<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 1, 2014 at 3:34 AM, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi!<div><br></div><div>Maybe you're facing this problem:</div><div><br></div><div><a href="https://bugs.launchpad.net/neutron/+bug/1252900" target="_blank">https://bugs.launchpad.net/neutron/+bug/1252900</a><br>
</div><div>
<br></div><div>I'm seeing that you tried to turn GRO off but, is your Network Node a Virtual Machine, or a physical server? If it is a VM (KVM), you'll need to turn GRO off at the hypervisor too.</div><div><br></div>
<div>Happy new year!</div><div><br></div><div>Cheers!</div><div>Thiago</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On 31 December 2013 13:55, sylecn <span dir="ltr"><<a href="mailto:sylecn@gmail.com" target="_blank">sylecn@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div>Hi all,<br><br></div>I have come across a network problem when doing network performance testing. From vm, wget a file on external network is much slower than expected.<br>
<br><div>The network is setup like this [1].<br>
<br></div><div>To download a file, packets travel from target server public ip to public ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node, via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a long chain.<br>
</div></div><div><br></div><div></div><div>Here are some numbers:<br></div><div>wget <a href="http://mytestserver/big-file.deb" target="_blank">http://mytestserver/big-file.deb</a><br><br></div><div>I use /etc/hosts to set mytestserver to the IP I want to test.<br>
<br></div><div> Download from <a href="http://10.20.165.2" target="_blank">10.20.165.2</a>: 47.2 MB/s (395Mbps) slow but acceptable.<br></div><div> Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps) way too slow.<br>
<br></div><div>I also tried the reverse: run web server on vm, and wget on <a href="http://10.20.165.2" target="_blank">10.20.165.2</a>:<br><br></div><div> Download from vm's floatingip (10.21.144.106) from outside: 105MB/s (880Mbps)<br>
<br></div><div>I have yet to test upload speed to outside network. Upload to <a href="http://10.20.165.0/24" target="_blank">10.20.165.0/24</a> and <a href="http://10.21.144.0/24" target="_blank">10.21.144.0/24</a> are both at 40+MB/s. Upload to public ip not tested yet.<br>
</div><br></div>Here are some raw iperf performance result:<br><br>with "GRO off" on br-ex and eth1:<br>| scenario | outgoing (from vm) | incoming (to vm) |<br>|----------------------------+--------------------+------------------|<br>
| vm to vm on same host | 3.16 Gbps | - |<br>| vm to vm on different host | 613 Mbps | - |<br>| vm to <a href="http://10.21.144.0/24" target="_blank">10.21.144.0/24</a> | 887 Mbps | 860 Mbps |<br>
| vm to <a href="http://10.20.165.0/24" target="_blank">10.20.165.0/24</a> | 888 Mbps | 818 Mbps |<br>| vm to <public ip> | 919 Mbps | 817 Mbps |<br><br>I'm using neutron with ovs plugin and GRE tunnels. There are no errors in neutron's log files, except<br>
<div><br>ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver<br><br></div><div>and similar which has always existed since I deploy openstack.<br><br></div><div>Things I have tried and did not resolve the problem:<br>
<br></div><div> - turn off GRO on neutron node br-ex/eth1 interface (kept off for all wget/scp tests)<br></div><div> - allow all ports in security groups<br></div><div> - drop iptables rules on neutron node<br></div><div>
- lower MTU to 1400 (This helped a little, all wget numbers above have MTU set to 1400 in vm)<br><br></div><div>I also suspected this nat rule is too slow:<br><br> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE<br>
<br></div><div>But test had proved me wrong. I have tested using 10.21.144.5 as the default gateway in a physical server and wget the same file from is lightning fast.<br><br></div><div>So is this download speed typical? If not, how can I isolate the problem and figure out what cause the slowness?<br>
<br></div><div>Thanks,<br>Yuanle<br></div><div><br>[1] <a href="http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png" target="_blank">http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png</a></div></div>
<br></div></div>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>