Open Stack

Hi James, have you tried setting the MTU to a lower number of bytes, 
instead of a higher-than-1500 setting? Say... 1454 instead of 1546?

Curious to see if that resolves the issue. If it does, then perhaps 
there is a path somewhere that had a <1546 PMTU?

-jay

On 10/02/2013 05:14 AM, James Page wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi Folks
>
> I'm seeing an odd direction performance issue with my Havana test rig
> which I'm struggling to debug; details:
>
> Ubuntu 12.04 with Linux 3.8 backports kernel, Havana Cloud Archive
> (currently Havana b3, OpenvSwitch 1.10.2), OpenvSwitch plugin with GRE
> overlay networks.
>
> I've configured the MTU's on all of the physical host network
> interfaces to 1546 to add capacity for the GRE network headers.
>
> Performance between instances within a single tenant network on
> different physical hosts is as I would expect (near 1GBps), but I see
> issues when data transits the Neutron L3 gateway - in the example
> below churel is a physical host on the same network as the layer 3
> gateway:
>
> ubuntu at churel:~$ scp hardware.dump 10.98.191.103:
> hardware.dump
>                                                100%   67MB   4.8MB/s
> 00:14
>
> ubuntu at churel:~$ scp 10.98.191.103:hardware.dump .
> hardware.dump
>                                                      100%   67MB
> 66.8MB/s   00:01
>
> As you can see, pushing data to the instance (via a floating ip
> 10.98.191.103) is painfully slow, whereas pulling the same data is
> x10+ faster (and closer to what I would expect).
>
> iperf confirms the same:
>
> ubuntu at churel:~$ iperf -c 10.98.191.103 -m
> - ------------------------------------------------------------
> Client connecting to 10.98.191.103, TCP port 5001
> TCP window size: 22.9 KByte (default)
> - ------------------------------------------------------------
> [  3] local 10.98.191.11 port 55330 connected with 10.98.191.103 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  60.8 MBytes  50.8 Mbits/sec
> [  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
>
> ubuntu at james-page-bastion:~$ iperf -c 10.98.191.11 -m
>
>
> - ------------------------------------------------------------
> Client connecting to 10.98.191.11, TCP port 5001
> TCP window size: 23.3 KByte (default)
> - ------------------------------------------------------------
> [  3] local 10.5.0.2 port 52190 connected with 10.98.191.11 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  1.07 GBytes   918 Mbits/sec
> [  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
>
>
> 918Mbit vs 50Mbits.
>
> I tcpdump'ed the traffic and I see alot of duplicate acks which makes
> me suspect some sort of packet fragmentation but its got me puzzled.
>
> Anyone have any ideas about how to debug this further? or has anyone
> seen anything like this before?
>
> Cheers
>
> James
>
>
> - --
> James Page
> Ubuntu and Debian Developer
> james.page at ubuntu.com
> jamespage at debian.org
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJSS+QSAAoJEL/srsug59jD8ZcQAKbZDVU8KKa7hsic7+ulqWQQ
> EFbq8Im5x4mQY7htIvIOM26BR0ktAO5luE7zMBXsA4AwPud1BQSGhw89/NvNhADT
> TLcGdQADsomeiBpJebzwUmvL/tYUoMDRA3O96mUn2pi0fySWbEuEgMDjDJ/ow23D
> Y7nEv0mItaZ4MBSI9RZcqsDUl7UbbdlGejSWhJcwp/127HMU9nYwWNz5UHJjsGZ1
> eITyv1WZH/dYPQ1SES41qD1WvkTBugopGJvptEyrcO62A+akGOvnqpsHgPECbLb+
> b/8rk8nB1HB74Wh+tQP4WRQCZYso15nB6ukIyIU24Qti2tXtXDdKwszEoblCwCT3
> YZJTERNOENURlUEFwgi6FNL+nZomSG0UJU6qqDGiUJkbSF7SwJm4y8/XRlJM2Ihn
> wyxFB0qe3YdMqgDLZn11GwCDqn3g11hYaocHNUyRaj/tgxhGKbOFvix5kz3I4V7T
> gd+sqUySMVd9wCRXBzDDhCuG9xf/QY2ZQxXzyfPJWd9svPh/O6osTSQzaI1eZl9/
> jVRejMAFr6Rl11GPKd3DYi32GXa896QELjBmJ9Kof0NDlCcDuUKpVeifIhcbQZZV
> sWyQmbb6Z/ypFV9xXiLRfH2fW2bAQQHgiQGvy9apoE78BWYdnsD8Q3Ekwag6lFqp
> yUwt/RcRXS1PbLG4EGFW
> =HTvW
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>