Open Stack

Hi
We are seeing a very weird issue with OVS 2.9 which is not seen in OVS 2.6. Basically the end symptom is from the neutron L3 agent  router namespace we cant ping the gateway in this case 10.86.67.1
Now on performing tcpdump tests we see the gateway responding to ARP the reply comes into the control/network node and is dropped by OVS in the qg- interface
We observed that when the ARP reply came back to OVS port it added the following entry
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=fa:16:3e:65:85:ad),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806)), packets:217, bytes:13888, used:0.329s, actions:drop

That drop rule states src mac=gateway mac(f0:25:72:ab:d4:c1), destination mac=qg-xxx interface drop the packet. We were first not sure why this was happening but when we inspected the arp response packet from gateway we noticed in this setup the packet from gatway was sent with cos /tos bit set to priority 5.

When we rewrote the packet on TOR to set cos/tos priority to 0 it worked fine.

Question is why does OVS 2.9 add a drop rule when it sees an ARP response with cos/tos priority set to 5.

Has anyone seen this before 2.6 with newton works for this use case and 2.9 with queens fails

Some info below



L3 Namespace
 ip netns exec qrouter-eee032f4-670f-4e43-8e83-e04cb23f00ae ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536<tel:65536> qdisc noqueue state UNKNOWN group default qlen 1000<tel:1000>
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
46: ha-df4ad8aa-18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<tel:1500> qdisc noqueue state UNKNOWN group default qlen 1000<tel:1000>
    link/ether fa:16:3e:eb:0c:21 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.5/18<http://169.254.192.5/18> brd 169.254.255.255<tel:169.254.255.255> scope global ha-df4ad8aa-18
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24<http://169.254.0.1/24> scope global ha-df4ad8aa-18
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feeb:c21/64 scope link
       valid_lft forever preferred_lft forever
47: qg-e5541f70-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500<tel:1500> qdisc noqueue state UNKNOWN group default qlen 1000<tel:1000>
    link/ether fa:16:3e:65:85:ad brd ff:ff:ff:ff:ff:ff
    inet 10.86.67.78/24<http://10.86.67.78/24> scope global qg-e5541f70-a5
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe65:85ad/64 scope link nodad
       valid_lft forever preferred_lft forever


Appctl Trace output
ovs_vswitch_15520 [root at BXB-AIO-1 /]# ovs-appctl ofproto/trace br-int in_port=2,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad
Flow: in_port=2,vlan_tci=0x0000,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x0000

bridge("br-int")
----------------
 0. in_port=2,vlan_tci=0x0000/0x1fff, priority 3, cookie 0x21589cb48848e7fb
    push_vlan:0x8100
    set_field:4102->vlan_vid
    goto_table:60
60. priority 3, cookie 0x21589cb48848e7fb
    NORMAL
     -> no learned MAC for destination, flooding

    bridge("br-prov")
    -----------------
         0. in_port=2, priority 2, cookie 0x7faae4a30960716f
            drop

bridge("br-inst")
-----------------
 0. in_port=2, priority 2, cookie 0x8b9b0311aedc1b0c
    drop

Final flow: in_port=2,dl_vlan=6,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x0000
Megaflow: recirc_id=0,eth,in_port=2,vlan_tci=0x0000/0x1fff,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x0000
Datapath actions: 9,push_vlan(vid=6,pcp=0),7
ovs_vswitch_15520 [root at BXB-AIO-1 /]#

ovs-dpctl dump-flows
ovs_vswitch_15520 [root at BXB-AIO-1 /]# ovs-dpctl dump-flows | grep f0:25:72:ab:d4:c1
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=ff:ff:ff:ff:ff:ff),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806),arp(sip=10.86.67.1,tip=10.86.67.77,op=1/0xff<http://10.86.67.1,tip=10.86.67.77,op=1/0xff>)), packets:3, bytes:192, used:8.199s, actions:1
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=fa:16:3e:65:85:ad),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806)), packets:217, bytes:13888, used:0.329s, actions:drop
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=ff:ff:ff:ff:ff:ff),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806),arp(sip=10.86.67.1,tip=10.86.67.45,op=1/0xff<http://10.86.67.1,tip=10.86.67.45,op=1/0xff>)), packets:3, bytes:192, used:9.778s, actions:1

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180823/32fa3504/attachment.html>