Open Stack

Hi all,

I have found the cause of the problem. When you use eth1 as br-ex where
eth1 is not connected to internet, and use an iptables NAT rule as software
gateway to internet, you need to turn off GRO on both eth0 and eth1.

After turning off GRO on eth0, download speed inside vm has become normal.

Thanks,
Yuanle


On Wed, Jan 1, 2014 at 1:29 PM, sylecn <sylecn at gmail.com> wrote:

>
> Hi Thiago,
>
> The Network Node is a physical server. GRO does affect network much. When
> GRO is on on the neutron node, from outside to VM bandwidth is about
> 60Mbps. When it is turned off, from outside to VM bandwidth is over
> 800Mbps. This is tested using iperf, physical network is 1Gbps.
>
> Happy new year to you too!
>
> Thanks,
> Yuanle
>
>
> On Wed, Jan 1, 2014 at 3:34 AM, Martinx - ジェームズ <thiagocmartinsc at gmail.com
> > wrote:
>
>> Hi!
>>
>> Maybe you're facing this problem:
>>
>> https://bugs.launchpad.net/neutron/+bug/1252900
>>
>> I'm seeing that you tried to turn GRO off but, is your Network Node a
>> Virtual Machine, or a physical server? If it is a VM (KVM), you'll need to
>> turn GRO off at the hypervisor too.
>>
>> Happy new year!
>>
>> Cheers!
>> Thiago
>>
>>
>> On 31 December 2013 13:55, sylecn <sylecn at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I have come across a network problem when doing network performance
>>> testing. From vm, wget a file on external network is much slower than
>>> expected.
>>>
>>> The network is setup like this [1].
>>>
>>> To download a file, packets travel from target server public ip to
>>> public ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron
>>> node, via neutron virtual router (l3-agent) to 10.20.144.100, then to
>>> virtual subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This
>>> is a long chain.
>>>
>>> Here are some numbers:
>>> wget http://mytestserver/big-file.deb
>>>
>>> I use /etc/hosts to set mytestserver to the IP I want to test.
>>>
>>>     Download from 10.20.165.2: 47.2 MB/s (395Mbps)
>>> slow but acceptable.
>>>     Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps)       way
>>> too slow.
>>>
>>> I also tried the reverse: run web server on vm, and wget on 10.20.165.2:
>>>
>>>     Download from vm's floatingip (10.21.144.106) from outside: 105MB/s
>>> (880Mbps)
>>>
>>> I have yet to test upload speed to outside network. Upload to
>>> 10.20.165.0/24 and 10.21.144.0/24 are both at 40+MB/s. Upload to public
>>> ip not tested yet.
>>>
>>> Here are some raw iperf performance result:
>>>
>>> with "GRO off" on br-ex and eth1:
>>> | scenario                   | outgoing (from vm) | incoming (to vm) |
>>> |----------------------------+--------------------+------------------|
>>> | vm to vm on same host      | 3.16 Gbps          | -                |
>>> | vm to vm on different host | 613 Mbps           | -                |
>>> | vm to 10.21.144.0/24       | 887 Mbps           | 860 Mbps         |
>>> | vm to 10.20.165.0/24       | 888 Mbps           | 818 Mbps         |
>>> | vm to <public ip>          | 919 Mbps           | 817 Mbps         |
>>>
>>> I'm using neutron with ovs plugin and GRE tunnels. There are no errors
>>> in neutron's log files, except
>>>
>>> ERROR neutron.common.legacy [-] Skipping unknown group key:
>>> firewall_driver
>>>
>>> and similar which has always existed since I deploy openstack.
>>>
>>> Things I have tried and did not resolve the problem:
>>>
>>>  - turn off GRO on neutron node br-ex/eth1 interface (kept off for all
>>> wget/scp tests)
>>>  - allow all ports in security groups
>>>  - drop iptables rules on neutron node
>>>  - lower MTU to 1400 (This helped a little, all wget numbers above have
>>> MTU set to 1400 in vm)
>>>
>>> I also suspected this nat rule is too slow:
>>>
>>>    iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
>>>
>>> But test had proved me wrong. I have tested using 10.21.144.5 as the
>>> default gateway in a physical server and wget the same file from  is
>>> lightning fast.
>>>
>>> So is this download speed typical? If not, how can I isolate the problem
>>> and figure out what cause the slowness?
>>>
>>> Thanks,
>>> Yuanle
>>>
>>> [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140103/05119c33/attachment.html>