[Openstack-operators] anyone using openvswitch 2.0.1 (ubuntu trusty) with GRE tunnels

George Shuklin george.shuklin at gmail.com
Sun Jan 19 19:50:45 UTC 2014


Never, EVER use any OVS < 1.10 in production.

Here simple proof:

hping3 -i u 100 any-floating-ip --rand-source

and networking node is dead. If hping3 happens inside instance, compute 
node is dead and so on.

Reason: OVS prior 1.11 can't use megaflow and any new port/ip address 
cause query from kernel to userspace. This is very slow process and 
ovs-switchd consume 100% cpu and cause huge packet loss. OVS 1.11+ can 
use asterisks (*) in their kernel flows (so called megaflow) so they can 
withstand flood.

PS That problem is not just DoS attacking. I saw (at XenServer large 
installation, which use OVS 1.9) when lots of legitimate small tcp 
sessions (popular browser-mmo-game-site) cause cripple on servers.

On 01/19/2014 11:56 AM, Robert Collins wrote:
> We've been seeing the sort of asymmetrically slow performance with
> openvswitch 1.10.2 on linux 3.11 in the ci cloud we're trying to push
> through to production at the moment.
>
> Based on reports of 1.11 fixing things, I looked for 1.11 packages but
> since trusty has 2.0.1 I figured it was easier to yank that in and see
> what happened.
>
> However, it's broken - I've got a few symptoms documented, though I
> don't know whats cause and what effect :).
>
> Firstly, outbound GRE packets are sent just fine. On a machine running
> 1.10.2, they are received and processed correctly.
>
> Inbound GRE packets are not received though.
> tcpdump shows them on the physical interface(eth2) and the local
> bridged (br-untagged) but they don't hit br-tun at all:
>
> ovs-ofctl dump-flows br-tun
> NXST_FLOW reply (xid=0x4):
>   cookie=0x0, duration=471.219s, table=0, n_packets=483, n_bytes=39986,
> idle_age=1, priority=1,in_port=1 actions=resubmit(,1)
>   cookie=0x0, duration=470.535s, table=0, n_packets=0, n_bytes=0,
> idle_age=470, priority=1,in_port=2 actions=resubmit(,2)
> ...
> note the n_packets=0 on in_port 2, which is the gre port:
> ...
>   2(gre-10.10.16.17): addr:92:07:f1:42:f3:a4
>       config:     0
>       state:      0
>       speed: 0 Mbps now, 0 Mbps max
> oddly but perhaps unrelated?, that port name is truncated -
>      Bridge br-tun
>          Port br-tun
>              Interface br-tun
>                  type: internal
>          Port "gre-10.10.16.175"
>              Interface "gre-10.10.16.175"
>                  type: gre
>                  options: {in_key=flow, local_ip="10.10.16.176",
> out_key=flow, remote_ip="10.10.16.175"}
>          Port patch-int
>              Interface patch-int
>                  type: patch
>                  options: {peer=patch-tun}
>
> The kernel datapath doesn't bring up the incoming flow - for instance,
> on 1.10.2 we'd see:
> # ovs-appctl dpif/dump-flows br-tun
> tunnel(tun_id=0x1,src=10.10.16.175,dst=10.10.16.176,tos=0x0,ttl=64,flags(key)),in_port(3),eth(src=fa:16:3e:c7:fd:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=10.0.0.1,tip=10.0.0.6,op=1,sha=fa:16:3e:c7:fd:70,tha=00:00:00:00:00:00),
> packets:3963, bytes:166446, used:0.756s,
> actions:push_vlan(vid=1,pcp=0),1,pop_vlan,8,16,14,push_vlan(vid=1,pcp=0),6,pop_vlan,12,10
> in_port(2),eth(src=56:96:98:5e:94:4a,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0,dst=255.255.255.255,proto=17,tos=0x10,ttl=128,frag=no),udp(src=68,dst=67),
> packets:0, bytes:0, used:4.610s, actions:drop
> #
>
> but on 2.0.1 we see:
> # ovs-appctl dpif/dump-flows br-tun
> #
>
> There's nothing in iptables-save to suggest we're filtering GRE (and
> in fact just replacing the openvswitch module without rebooting or
> running iptables commands).
>
> I'm not sure how/where the incoming GRE packets are handled - I
> suspect it's in-kernel and somewhat inaccessible for debugging...
>
> -Rob
>




More information about the OpenStack-operators mailing list