[Openstack] Directional network performance issues with Neutron + OpenvSwitch

Robert Collins robertc at robertcollins.net
Thu Oct 3 01:28:07 UTC 2013


I believe it's still needed: upstream kernel have pushed back against
the modules it provides, but neutron needs them to deliver the gre
tunnels.

-Rob

On 3 October 2013 13:15, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
> Hi James,
>
> Let me ask you something...
>
> Are you using the package `openvswitch-datapath-dkms' from Havana Ubuntu
> Cloud Archive with Linux 3.8?
>
> I am unable to compile that module on top of Ubuntu 12.04.3 (with Linux 3.8)
> and I'm wondering if it is still required or not...
>
> Thanks!
> Thiago
>
>
> On 2 October 2013 06:14, James Page <james.page at ubuntu.com> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Hi Folks
>>
>> I'm seeing an odd direction performance issue with my Havana test rig
>> which I'm struggling to debug; details:
>>
>> Ubuntu 12.04 with Linux 3.8 backports kernel, Havana Cloud Archive
>> (currently Havana b3, OpenvSwitch 1.10.2), OpenvSwitch plugin with GRE
>> overlay networks.
>>
>> I've configured the MTU's on all of the physical host network
>> interfaces to 1546 to add capacity for the GRE network headers.
>>
>> Performance between instances within a single tenant network on
>> different physical hosts is as I would expect (near 1GBps), but I see
>> issues when data transits the Neutron L3 gateway - in the example
>> below churel is a physical host on the same network as the layer 3
>> gateway:
>>
>> ubuntu at churel:~$ scp hardware.dump 10.98.191.103:
>> hardware.dump
>>                                               100%   67MB   4.8MB/s
>> 00:14
>>
>> ubuntu at churel:~$ scp 10.98.191.103:hardware.dump .
>> hardware.dump
>>                                                     100%   67MB
>> 66.8MB/s   00:01
>>
>> As you can see, pushing data to the instance (via a floating ip
>> 10.98.191.103) is painfully slow, whereas pulling the same data is
>> x10+ faster (and closer to what I would expect).
>>
>> iperf confirms the same:
>>
>> ubuntu at churel:~$ iperf -c 10.98.191.103 -m
>> - ------------------------------------------------------------
>> Client connecting to 10.98.191.103, TCP port 5001
>> TCP window size: 22.9 KByte (default)
>> - ------------------------------------------------------------
>> [  3] local 10.98.191.11 port 55330 connected with 10.98.191.103 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0-10.0 sec  60.8 MBytes  50.8 Mbits/sec
>> [  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
>>
>> ubuntu at james-page-bastion:~$ iperf -c 10.98.191.11 -m
>>
>>
>> - ------------------------------------------------------------
>> Client connecting to 10.98.191.11, TCP port 5001
>> TCP window size: 23.3 KByte (default)
>> - ------------------------------------------------------------
>> [  3] local 10.5.0.2 port 52190 connected with 10.98.191.11 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0-10.0 sec  1.07 GBytes   918 Mbits/sec
>> [  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
>>
>>
>> 918Mbit vs 50Mbits.
>>
>> I tcpdump'ed the traffic and I see alot of duplicate acks which makes
>> me suspect some sort of packet fragmentation but its got me puzzled.
>>
>> Anyone have any ideas about how to debug this further? or has anyone
>> seen anything like this before?
>>
>> Cheers
>>
>> James
>>
>>
>> - --
>> James Page
>> Ubuntu and Debian Developer
>> james.page at ubuntu.com
>> jamespage at debian.org
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.14 (GNU/Linux)
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJSS+QSAAoJEL/srsug59jD8ZcQAKbZDVU8KKa7hsic7+ulqWQQ
>> EFbq8Im5x4mQY7htIvIOM26BR0ktAO5luE7zMBXsA4AwPud1BQSGhw89/NvNhADT
>> TLcGdQADsomeiBpJebzwUmvL/tYUoMDRA3O96mUn2pi0fySWbEuEgMDjDJ/ow23D
>> Y7nEv0mItaZ4MBSI9RZcqsDUl7UbbdlGejSWhJcwp/127HMU9nYwWNz5UHJjsGZ1
>> eITyv1WZH/dYPQ1SES41qD1WvkTBugopGJvptEyrcO62A+akGOvnqpsHgPECbLb+
>> b/8rk8nB1HB74Wh+tQP4WRQCZYso15nB6ukIyIU24Qti2tXtXDdKwszEoblCwCT3
>> YZJTERNOENURlUEFwgi6FNL+nZomSG0UJU6qqDGiUJkbSF7SwJm4y8/XRlJM2Ihn
>> wyxFB0qe3YdMqgDLZn11GwCDqn3g11hYaocHNUyRaj/tgxhGKbOFvix5kz3I4V7T
>> gd+sqUySMVd9wCRXBzDDhCuG9xf/QY2ZQxXzyfPJWd9svPh/O6osTSQzaI1eZl9/
>> jVRejMAFr6Rl11GPKd3DYi32GXa896QELjBmJ9Kof0NDlCcDuUKpVeifIhcbQZZV
>> sWyQmbb6Z/ypFV9xXiLRfH2fW2bAQQHgiQGvy9apoE78BWYdnsD8Q3Ekwag6lFqp
>> yUwt/RcRXS1PbLG4EGFW
>> =HTvW
>> -----END PGP SIGNATURE-----
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud




More information about the Openstack mailing list