Open Stack

I have an odd issue that seems to just be affecting one private
network for one tenant, though I saw a similar thing on a different
project network recently which I 'fixed' by rebooting the hypervisor.
Since this has now (maybe) happened twice I figure I should try to
understand what it is.

Given the following four VMs on 4 different hypervisors

vm1 on Hypervisor1
vm2 on Hypervisor2
vm3 on Hypervisor3
-------------------
vm4 on Hypervisor4


vm1 -> vm3 talk fine among themselves but none to 4

examining ping traffic transiting from vm1-vm4 I can see arp requests
and responses at vm4 and GRE encapsulated ARP responses on
Hypervisor1's physical interface.

They look the same to me (same ecap id) coming in as the working vms
traffic, but they never make it to the qvo device which is before
iptables sec_group rules are applied at the tap device.

attempting to tare down and recreate this resuls in the same first 3
work last one doesn't split (possibly becuase scheduler puts them in
the same place? haven't checked) 

ovs-vsctl -- set Bridge br-int mirrors=@m  -- --id=@snooper2 get Port snooper2  -- --id=@gre-801e0347 get Port gre-801e0347 -- --id=@m create Mirror name=mymirror select-dst-port=@gre-801e0347 select-src-port=@gre-801e0347 output-port=@snooper2

tcpdump -i snooper2 

Only sees ARP requests but no response, what's broken if I can see GRE
encap ARP responses on physical interface but not on gre-<hex>
interface?  And why is it not broken for all tunnels endpoints?

Oddly if I boot a 5th VM on a 5th hypervisor it can talk to 4 but not 1-3 ...

hypervisors are Ubuntu 14.04 running Mitaka from cloud archive w/
xenial-lts kernels (4.4.0)

-Jon

--