Open Stack

Hi all,

I have come across a network problem when doing network performance
testing. From vm, wget a file on external network is much slower than
expected.

The network is setup like this [1].

To download a file, packets travel from target server public ip to public
ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node,
via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual
subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a
long chain.

Here are some numbers:
wget http://mytestserver/big-file.deb

I use /etc/hosts to set mytestserver to the IP I want to test.

    Download from 10.20.165.2: 47.2 MB/s (395Mbps)                  slow
but acceptable.
    Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps)       way too
slow.

I also tried the reverse: run web server on vm, and wget on 10.20.165.2:

    Download from vm's floatingip (10.21.144.106) from outside: 105MB/s
(880Mbps)

I have yet to test upload speed to outside network. Upload to 10.20.165.0/24and
10.21.144.0/24 are both at 40+MB/s. Upload to public ip not tested yet.

Here are some raw iperf performance result:

with "GRO off" on br-ex and eth1:
| scenario                   | outgoing (from vm) | incoming (to vm) |
|----------------------------+--------------------+------------------|
| vm to vm on same host      | 3.16 Gbps          | -                |
| vm to vm on different host | 613 Mbps           | -                |
| vm to 10.21.144.0/24       | 887 Mbps           | 860 Mbps         |
| vm to 10.20.165.0/24       | 888 Mbps           | 818 Mbps         |
| vm to <public ip>          | 919 Mbps           | 817 Mbps         |

I'm using neutron with ovs plugin and GRE tunnels. There are no errors in
neutron's log files, except

ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver

and similar which has always existed since I deploy openstack.

Things I have tried and did not resolve the problem:

 - turn off GRO on neutron node br-ex/eth1 interface (kept off for all
wget/scp tests)
 - allow all ports in security groups
 - drop iptables rules on neutron node
 - lower MTU to 1400 (This helped a little, all wget numbers above have MTU
set to 1400 in vm)

I also suspected this nat rule is too slow:

   iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

But test had proved me wrong. I have tested using 10.21.144.5 as the
default gateway in a physical server and wget the same file from  is
lightning fast.

So is this download speed typical? If not, how can I isolate the problem
and figure out what cause the slowness?

Thanks,
Yuanle

[1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131231/ad60630f/attachment.html>