Open Stack

We've been seeing the sort of asymmetrically slow performance with
openvswitch 1.10.2 on linux 3.11 in the ci cloud we're trying to push
through to production at the moment.

Based on reports of 1.11 fixing things, I looked for 1.11 packages but
since trusty has 2.0.1 I figured it was easier to yank that in and see
what happened.

However, it's broken - I've got a few symptoms documented, though I
don't know whats cause and what effect :).

Firstly, outbound GRE packets are sent just fine. On a machine running
1.10.2, they are received and processed correctly.

Inbound GRE packets are not received though.
tcpdump shows them on the physical interface(eth2) and the local
bridged (br-untagged) but they don't hit br-tun at all:

ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=471.219s, table=0, n_packets=483, n_bytes=39986,
idle_age=1, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0x0, duration=470.535s, table=0, n_packets=0, n_bytes=0,
idle_age=470, priority=1,in_port=2 actions=resubmit(,2)
...
note the n_packets=0 on in_port 2, which is the gre port:
...
 2(gre-10.10.16.17): addr:92:07:f1:42:f3:a4
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
oddly but perhaps unrelated?, that port name is truncated -
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "gre-10.10.16.175"
            Interface "gre-10.10.16.175"
                type: gre
                options: {in_key=flow, local_ip="10.10.16.176",
out_key=flow, remote_ip="10.10.16.175"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}

The kernel datapath doesn't bring up the incoming flow - for instance,
on 1.10.2 we'd see:
# ovs-appctl dpif/dump-flows br-tun
tunnel(tun_id=0x1,src=10.10.16.175,dst=10.10.16.176,tos=0x0,ttl=64,flags(key)),in_port(3),eth(src=fa:16:3e:c7:fd:70,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=10.0.0.1,tip=10.0.0.6,op=1,sha=fa:16:3e:c7:fd:70,tha=00:00:00:00:00:00),
packets:3963, bytes:166446, used:0.756s,
actions:push_vlan(vid=1,pcp=0),1,pop_vlan,8,16,14,push_vlan(vid=1,pcp=0),6,pop_vlan,12,10
in_port(2),eth(src=56:96:98:5e:94:4a,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0,dst=255.255.255.255,proto=17,tos=0x10,ttl=128,frag=no),udp(src=68,dst=67),
packets:0, bytes:0, used:4.610s, actions:drop
#

but on 2.0.1 we see:
# ovs-appctl dpif/dump-flows br-tun
#

There's nothing in iptables-save to suggest we're filtering GRE (and
in fact just replacing the openvswitch module without rebooting or
running iptables commands).

I'm not sure how/where the incoming GRE packets are handled - I
suspect it's in-kernel and somewhat inaccessible for debugging...

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud