Open Stack

Hi!

Maybe you're facing this problem:

https://bugs.launchpad.net/neutron/+bug/1252900

I'm seeing that you tried to turn GRO off but, is your Network Node a
Virtual Machine, or a physical server? If it is a VM (KVM), you'll need to
turn GRO off at the hypervisor too.

Happy new year!

Cheers!
Thiago


On 31 December 2013 13:55, sylecn <sylecn at gmail.com> wrote:

> Hi all,
>
> I have come across a network problem when doing network performance
> testing. From vm, wget a file on external network is much slower than
> expected.
>
> The network is setup like this [1].
>
> To download a file, packets travel from target server public ip to public
> ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node,
> via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual
> subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a
> long chain.
>
> Here are some numbers:
> wget http://mytestserver/big-file.deb
>
> I use /etc/hosts to set mytestserver to the IP I want to test.
>
>     Download from 10.20.165.2: 47.2 MB/s (395Mbps)                  slow
> but acceptable.
>     Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps)       way
> too slow.
>
> I also tried the reverse: run web server on vm, and wget on 10.20.165.2:
>
>     Download from vm's floatingip (10.21.144.106) from outside: 105MB/s
> (880Mbps)
>
> I have yet to test upload speed to outside network. Upload to
> 10.20.165.0/24 and 10.21.144.0/24 are both at 40+MB/s. Upload to public
> ip not tested yet.
>
> Here are some raw iperf performance result:
>
> with "GRO off" on br-ex and eth1:
> | scenario                   | outgoing (from vm) | incoming (to vm) |
> |----------------------------+--------------------+------------------|
> | vm to vm on same host      | 3.16 Gbps          | -                |
> | vm to vm on different host | 613 Mbps           | -                |
> | vm to 10.21.144.0/24       | 887 Mbps           | 860 Mbps         |
> | vm to 10.20.165.0/24       | 888 Mbps           | 818 Mbps         |
> | vm to <public ip>          | 919 Mbps           | 817 Mbps         |
>
> I'm using neutron with ovs plugin and GRE tunnels. There are no errors in
> neutron's log files, except
>
> ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
>
> and similar which has always existed since I deploy openstack.
>
> Things I have tried and did not resolve the problem:
>
>  - turn off GRO on neutron node br-ex/eth1 interface (kept off for all
> wget/scp tests)
>  - allow all ports in security groups
>  - drop iptables rules on neutron node
>  - lower MTU to 1400 (This helped a little, all wget numbers above have
> MTU set to 1400 in vm)
>
> I also suspected this nat rule is too slow:
>
>    iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
>
> But test had proved me wrong. I have tested using 10.21.144.5 as the
> default gateway in a physical server and wget the same file from  is
> lightning fast.
>
> So is this download speed typical? If not, how can I isolate the problem
> and figure out what cause the slowness?
>
> Thanks,
> Yuanle
>
> [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131231/190cb9f6/attachment.html>