<div dir="ltr"><div><div>Hi all,<br><br>I'm having some trouble with a Neutron
(icehouse on Ubuntu 14.04) configuration and was hoping somebody could
shed some light on it. My network node is seeing DHCP requests and
trying to respond to them, but they're getting lost in my openvswitch
network. This behavior is inconsistent (some VMs get addresses and some
don't), but it persists across VM reboots (that is, if a VM gets an IP
once, it appears to get one every time it reboots and if it fails the
first time, it will fail after every reboot). <br><br>The setup is as follows:<br><br></div>External network: <a href="http://10.10.1.0/24" target="_blank">10.10.1.0/24</a><br></div><div>Internal network: <a href="http://192.168.99.0/24" target="_blank">192.168.99.0/24</a><br></div><div>GRE tunnels<br></div><div>Four compute nodes: 192.168.1.10[4,5,6,7]<br></div><div>3 control/network nodes (HA using Pacemaker): 192.168.1.10[0,2,3]<br></div><div>A mysql cluster for the database (Percona XtraDB)<br></div><br><div>On my compute node, ovs-vsctl show gives:<br><br> Bridge br-int<br> fail_mode: secure<br> Port br-int<br> Interface br-int<br> type: internal<br> Port "qvoec5db497-3b"<br> tag: 1<br> Interface "qvoec5db497-3b"<br> Port "em2"<br> Interface "em2"<br> Port "snoop0"<br> Interface "snoop0"<br> Port patch-tun<br> Interface patch-tun<br> type: patch<br> options: {peer=patch-int}<br> Port int-br-ex<br> Interface int-br-ex<br> Bridge br-ex<br> Port br-ex<br> Interface br-ex<br> type: internal<br> Port "em1"<br> Interface "em1"<br> Port phy-br-ex<br> Interface phy-br-ex<br> Bridge br-tun<br> Port "gre-c0a86369"<br> Interface "gre-c0a86369"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.105"}<br> Port "gre-c0a86364"<br> Interface "gre-c0a86364"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.100"}<br> Port patch-int<br> Interface patch-int<br> type: patch<br> options: {peer=patch-tun}<br> Port br-tun<br> Interface br-tun<br> type: internal<br> Port "gre-c0a86366"<br> Interface "gre-c0a86366"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.102"}<br> Port "gre-c0a86367"<br> Interface "gre-c0a86367"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.103"}<br> Port "gre-c0a86368"<br> Interface "gre-c0a86368"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.104"}<br> Port "gre-c0a8636a"<br> Interface "gre-c0a8636a"<br> type: gre<br> options: {in_key=flow, local_ip="192.168.99.107", out_key=flow, remote_ip="192.168.99.106"}<br> ovs_version: "2.0.2"<br><br> <br></div><div>With a VM on 192.168.99.107, I can tap into the various ports on the compute node and make the following observations:<br><br></div><div>1) Requests are coming out of qvoec5db497-3b.<br></div><div>2) Replies are coming back over gre-c0a86364 (sensible, as the DHCP server is on 192.168.99.100).<br></div><div>3) I don't see requests on gre-c0a86364, and I don't see replies on
qvoec5db497-3b. Clearly the request is going out over the GRE tunnel,
though, because a reply is coming back. I'm using the following script
to observe the ports, so there may be a problem wiht my observations
here:<br><br>#!/bin/sh -x<br><br>bridge=$1<br>port=$2<br>ifname=snoop0<br><br>ovs-vsctl clear Bridge $bridge mirrors<br>ovs-vsctl del-port $ifname<br>ip link add name $ifname type dummy<br>ip link set dev $ifname up<br>ovs-vsctl add-port $bridge $ifname<br><br>ovs-vsctl -- set Bridge $bridge mirrors=@m \<br>-- --id=@$ifname get Port $ifname \<br>-- --id=@$port get Port $port \<br>-- --id=@m create Mirror name=m-$ifname select-dst-port=@$port select-src-port=@$port output-port=@$ifname<br><br></div><div>On the compute node, iptables -S is:<br><br>root@compute3:/home/mdavis# iptables -S<br>-P INPUT ACCEPT<br>-P FORWARD ACCEPT<br>-P OUTPUT ACCEPT<br>-N neutron-filter-top<br>-N neutron-openvswi-FORWARD<br>-N neutron-openvswi-INPUT<br>-N neutron-openvswi-OUTPUT<br>-N neutron-openvswi-iec5db497-3<br>-N neutron-openvswi-local<br>-N neutron-openvswi-oec5db497-3<br>-N neutron-openvswi-sec5db497-3<br>-N neutron-openvswi-sg-chain<br>-N neutron-openvswi-sg-fallback<br>-A INPUT -j neutron-openvswi-INPUT<br>-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT<br>-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT<br>-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT<br>-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT<br>-A FORWARD -j neutron-filter-top<br>-A FORWARD -j neutron-openvswi-FORWARD<br>-A FORWARD -d <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT<br>-A FORWARD -s <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> -i virbr0 -j ACCEPT<br>-A FORWARD -i virbr0 -o virbr0 -j ACCEPT<br>-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable<br>-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable<br>-A OUTPUT -j neutron-filter-top<br>-A OUTPUT -j neutron-openvswi-OUTPUT<br>-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT<br>-A neutron-filter-top -j neutron-openvswi-local<br>-A neutron-openvswi-FORWARD -m physdev --physdev-out tapec5db497-3b --physdev-is-bridged -j neutron-openvswi-sg-chain<br>-A neutron-openvswi-FORWARD -m physdev --physdev-in tapec5db497-3b --physdev-is-bridged -j neutron-openvswi-sg-chain<br>-A neutron-openvswi-INPUT -m physdev --physdev-in tapec5db497-3b --physdev-is-bridged -j neutron-openvswi-oec5db497-3<br>-A neutron-openvswi-iec5db497-3 -m state --state INVALID -j DROP<br>-A neutron-openvswi-iec5db497-3 -m state --state RELATED,ESTABLISHED -j RETURN<br>-A neutron-openvswi-iec5db497-3 -p tcp -m tcp --dport 443 -j RETURN<br>-A neutron-openvswi-iec5db497-3 -p tcp -m tcp --dport 22 -j RETURN<br>-A neutron-openvswi-iec5db497-3 -p tcp -m tcp --dport 80 -j RETURN<br>-A neutron-openvswi-iec5db497-3 -p icmp -j RETURN<br>-A neutron-openvswi-iec5db497-3 -s <a href="http://192.168.1.3/32" target="_blank">192.168.1.3/32</a> -p udp -m udp --sport 67 --dport 68 -j RETURN<br>-A neutron-openvswi-iec5db497-3 -j neutron-openvswi-sg-fallback<br>-A neutron-openvswi-oec5db497-3 -p udp -m udp --sport 68 --dport 67 -j RETURN<br>-A neutron-openvswi-oec5db497-3 -j neutron-openvswi-sec5db497-3<br>-A neutron-openvswi-oec5db497-3 -p udp -m udp --sport 67 --dport 68 -j DROP<br>-A neutron-openvswi-oec5db497-3 -m state --state INVALID -j DROP<br>-A neutron-openvswi-oec5db497-3 -m state --state RELATED,ESTABLISHED -j RETURN<br>-A neutron-openvswi-oec5db497-3 -j RETURN<br>-A neutron-openvswi-oec5db497-3 -j neutron-openvswi-sg-fallback<br>-A neutron-openvswi-sec5db497-3 -s <a href="http://192.168.1.15/32" target="_blank">192.168.1.15/32</a> -m mac --mac-source FA:16:3E:AD:45:3A -j RETURN<br>-A neutron-openvswi-sec5db497-3 -j DROP<br>-A neutron-openvswi-sg-chain -m physdev --physdev-out tapec5db497-3b --physdev-is-bridged -j neutron-openvswi-iec5db497-3<br>-A neutron-openvswi-sg-chain -m physdev --physdev-in tapec5db497-3b --physdev-is-bridged -j neutron-openvswi-oec5db497-3<br>-A neutron-openvswi-sg-chain -j ACCEPT<br>-A neutron-openvswi-sg-fallback -j DROP<br><br></div><div>Open vswitch flows are:<br><br>root@compute3:/home/mdavis# ovs-ofctl dump-flows br-int<br>NXST_FLOW reply (xid=0x4):<br> cookie=0x0, duration=6721.745s, table=0, n_packets=29791, n_bytes=5249622, idle_age=1, priority=2,in_port=4 actions=drop<br> cookie=0x0, duration=6722.49s, table=0, n_packets=1620378120, n_bytes=487782754336, idle_age=0, priority=1 actions=NORMAL<br> cookie=0x0, duration=6722.445s, table=22, n_packets=0, n_bytes=0, idle_age=6722, priority=0 actions=drop<br>root@compute3:/home/mdavis# ovs-ofctl dump-flows br-tun<br>NXST_FLOW reply (xid=0x4):<br> cookie=0x0, duration=6722.853s, table=0, n_packets=41, n_bytes=11814, idle_age=3754, priority=1,in_port=7 actions=resubmit(,2)<br> cookie=0x0, duration=6722.997s, table=0, n_packets=0, n_bytes=0, idle_age=6722, priority=1,in_port=3 actions=resubmit(,2)<br> cookie=0x0,
duration=6722.566s, table=0, n_packets=8488479, n_bytes=2950926206,
idle_age=3786, priority=1,in_port=6 actions=resubmit(,2)<br> cookie=0x0,
duration=6722.707s, table=0, n_packets=1399703, n_bytes=336420637,
idle_age=3717, priority=1,in_port=5 actions=resubmit(,2)<br> cookie=0x0,
duration=6723.95s, table=0, n_packets=186932608, n_bytes=46508961076,
idle_age=0, priority=1,in_port=1 actions=resubmit(,1)<br> cookie=0x0, duration=6723.132s, table=0, n_packets=0, n_bytes=0, idle_age=6723, priority=1,in_port=4 actions=resubmit(,2)<br> cookie=0x0,
duration=6723.273s, table=0, n_packets=197016168, n_bytes=69098476729,
idle_age=0, priority=1,in_port=2 actions=resubmit(,2)<br> cookie=0x0, duration=6723.905s, table=0, n_packets=0, n_bytes=0, idle_age=6723, priority=0 actions=drop<br> cookie=0x0,
duration=6723.861s, table=1, n_packets=2074, n_bytes=737325,
idle_age=390, priority=1,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00
actions=resubmit(,20)<br> cookie=0x0, duration=6723.814s, table=1,
n_packets=186930534, n_bytes=46508223751, idle_age=0,
priority=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00
actions=resubmit(,21)<br> cookie=0x0, duration=3901.865s, table=2,
n_packets=189487661, n_bytes=66331091024, idle_age=0,
priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10)<br> cookie=0x0, duration=6723.766s, table=2, n_packets=17416730, n_bytes=<a href="tel:6054744362" value="+16054744362" target="_blank">6054744362</a>, idle_age=4100, priority=0 actions=drop<br> cookie=0x0, duration=6723.724s, table=3, n_packets=0, n_bytes=0, idle_age=6723, priority=0 actions=drop<br> cookie=0x0,
duration=6723.683s, table=10, n_packets=189487661, n_bytes=66331091024,
idle_age=0, priority=1
actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1<br> cookie=0x0,
duration=3897.748s, table=20, n_packets=0, n_bytes=0, hard_timeout=300,
idle_age=3897, hard_age=0,
priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:c7:1d:3d
actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2<br> cookie=0x0, duration=6723.643s, table=20, n_packets=1, n_bytes=384, idle_age=3897, priority=0 actions=resubmit(,21)<br> cookie=0x0,
duration=3901.911s, table=21, n_packets=166399667, n_bytes=39541368007,
idle_age=0, dl_vlan=1
actions=strip_vlan,set_tunnel:0x1,output:4,output:3,output:2,output:6,output:7,output:5<br> cookie=0x0, duration=6723.594s, table=21, n_packets=20530868, n_bytes=6966856128, idle_age=3456, priority=0 actions=drop<br>root@compute3:/home/mdavis# ovs-ofctl dump-flows br-ex<br>NXST_FLOW reply (xid=0x4):<br> cookie=0x0,
duration=6730.347s, table=0, n_packets=180379513, n_bytes=43740118754,
idle_age=1, priority=2,in_port=3 actions=drop<br> cookie=0x0, duration=6730.867s, table=0, n_packets=470149, n_bytes=81654388, idle_age=0, priority=1 actions=NORMAL<br><br></div><div>My ml2_conf.ini for the node is:<br><br>[ml2]<br>type_drivers = flat,gre<br><br>tenant_network_types = flat,gre<br><br>mechanism_drivers = openvswitch<br><br>[ml2_type_flat]<br><br>[ml2_type_vlan]<br>network_vlan_ranges = phys_int:1000:1023,phys_ex:2000:2023<br><br>[ml2_type_gre]<br>tunnel_id_ranges = 1:1000<br><br>[securitygroup]<br>firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver<br>enable_security_group = True<br>enable_ipset = True<br><br>[ovs]<br>bridge_mappings = external:br-ex<br>local_ip = 192.168.99.107<br>tunnel_type = gre<br>enable_tunneling = True<br><br></div><div>Any thoughts as to what I may be doing wrong?<br><br>Thanks,<br><br></div>-Matt Davis</div>