Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Folks

I'm seeing an odd direction performance issue with my Havana test rig
which I'm struggling to debug; details:

Ubuntu 12.04 with Linux 3.8 backports kernel, Havana Cloud Archive
(currently Havana b3, OpenvSwitch 1.10.2), OpenvSwitch plugin with GRE
overlay networks.

I've configured the MTU's on all of the physical host network
interfaces to 1546 to add capacity for the GRE network headers.

Performance between instances within a single tenant network on
different physical hosts is as I would expect (near 1GBps), but I see
issues when data transits the Neutron L3 gateway - in the example
below churel is a physical host on the same network as the layer 3
gateway:

ubuntu at churel:~$ scp hardware.dump 10.98.191.103:
hardware.dump
                                              100%   67MB   4.8MB/s
00:14

ubuntu at churel:~$ scp 10.98.191.103:hardware.dump .
hardware.dump
                                                    100%   67MB
66.8MB/s   00:01

As you can see, pushing data to the instance (via a floating ip
10.98.191.103) is painfully slow, whereas pulling the same data is
x10+ faster (and closer to what I would expect).

iperf confirms the same:

ubuntu at churel:~$ iperf -c 10.98.191.103 -m
- ------------------------------------------------------------
Client connecting to 10.98.191.103, TCP port 5001
TCP window size: 22.9 KByte (default)
- ------------------------------------------------------------
[  3] local 10.98.191.11 port 55330 connected with 10.98.191.103 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  60.8 MBytes  50.8 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

ubuntu at james-page-bastion:~$ iperf -c 10.98.191.11 -m


- ------------------------------------------------------------
Client connecting to 10.98.191.11, TCP port 5001
TCP window size: 23.3 KByte (default)
- ------------------------------------------------------------
[  3] local 10.5.0.2 port 52190 connected with 10.98.191.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.07 GBytes   918 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


918Mbit vs 50Mbits.

I tcpdump'ed the traffic and I see alot of duplicate acks which makes
me suspect some sort of packet fragmentation but its got me puzzled.

Anyone have any ideas about how to debug this further? or has anyone
seen anything like this before?

Cheers

James


- -- 
James Page
Ubuntu and Debian Developer
james.page at ubuntu.com
jamespage at debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJSS+QSAAoJEL/srsug59jD8ZcQAKbZDVU8KKa7hsic7+ulqWQQ
EFbq8Im5x4mQY7htIvIOM26BR0ktAO5luE7zMBXsA4AwPud1BQSGhw89/NvNhADT
TLcGdQADsomeiBpJebzwUmvL/tYUoMDRA3O96mUn2pi0fySWbEuEgMDjDJ/ow23D
Y7nEv0mItaZ4MBSI9RZcqsDUl7UbbdlGejSWhJcwp/127HMU9nYwWNz5UHJjsGZ1
eITyv1WZH/dYPQ1SES41qD1WvkTBugopGJvptEyrcO62A+akGOvnqpsHgPECbLb+
b/8rk8nB1HB74Wh+tQP4WRQCZYso15nB6ukIyIU24Qti2tXtXDdKwszEoblCwCT3
YZJTERNOENURlUEFwgi6FNL+nZomSG0UJU6qqDGiUJkbSF7SwJm4y8/XRlJM2Ihn
wyxFB0qe3YdMqgDLZn11GwCDqn3g11hYaocHNUyRaj/tgxhGKbOFvix5kz3I4V7T
gd+sqUySMVd9wCRXBzDDhCuG9xf/QY2ZQxXzyfPJWd9svPh/O6osTSQzaI1eZl9/
jVRejMAFr6Rl11GPKd3DYi32GXa896QELjBmJ9Kof0NDlCcDuUKpVeifIhcbQZZV
sWyQmbb6Z/ypFV9xXiLRfH2fW2bAQQHgiQGvy9apoE78BWYdnsD8Q3Ekwag6lFqp
yUwt/RcRXS1PbLG4EGFW
=HTvW
-----END PGP SIGNATURE-----