Open Stack

On Wed, Nov 09, 2016 at 10:18:25AM -0500, Mohammed Naser wrote:
:Have you tried turning off all agents, shut off all VMs on that node, destroy br-int and be-tun, start neutron agents and try again?

Actually just figured out the problem and it's me :)

we have iptables rules on the hypervisor eth0
interfaces (with gre endpoint ips).  One of the allow rules isn't
quite right so that's where the traffic is dropping

Since these rules are not "OpenStack" managed I wasn't thinking about
them in my diagnostic tracing.

Oops,
-Jon

:
:Sent from my iPhone
:
:> On Nov 9, 2016, at 9:51 AM, Jonathan D. Proulx <jon at csail.mit.edu> wrote:
:> 
:> 
:> Also I have restarted openvswitch-agent on both sides of the broken
:> link and it made no difference...
:> 
:> On Tue, Nov 08, 2016 at 05:43:27PM -0500, Jonathan Proulx wrote:
:> :
:> :I have an odd issue that seems to just be affecting one private
:> :network for one tenant, though I saw a similar thing on a different
:> :project network recently which I 'fixed' by rebooting the hypervisor.
:> :Since this has now (maybe) happened twice I figure I should try to
:> :understand what it is.
:> :
:> :Given the following four VMs on 4 different hypervisors
:> :
:> :vm1 on Hypervisor1
:> :vm2 on Hypervisor2
:> :vm3 on Hypervisor3
:> :-------------------
:> :vm4 on Hypervisor4
:> :
:> :
:> :vm1 -> vm3 talk fine among themselves but none to 4
:> :
:> :examining ping traffic transiting from vm1-vm4 I can see arp requests
:> :and responses at vm4 and GRE encapsulated ARP responses on
:> :Hypervisor1's physical interface.
:> :
:> :They look the same to me (same ecap id) coming in as the working vms
:> :traffic, but they never make it to the qvo device which is before
:> :iptables sec_group rules are applied at the tap device.
:> :
:> :attempting to tare down and recreate this resuls in the same first 3
:> :work last one doesn't split (possibly becuase scheduler puts them in
:> :the same place? haven't checked) 
:> :
:> :ovs-vsctl -- set Bridge br-int mirrors=@m  -- --id=@snooper2 get Port snooper2  -- --id=@gre-801e0347 get Port gre-801e0347 -- --id=@m create Mirror name=mymirror select-dst-port=@gre-801e0347 select-src-port=@gre-801e0347 output-port=@snooper2
:> :
:> :tcpdump -i snooper2 
:> :
:> :Only sees ARP requests but no response, what's broken if I can see GRE
:> :encap ARP responses on physical interface but not on gre-<hex>
:> :interface?  And why is it not broken for all tunnels endpoints?
:> :
:> :Oddly if I boot a 5th VM on a 5th hypervisor it can talk to 4 but not 1-3 ...
:> :
:> :hypervisors are Ubuntu 14.04 running Mitaka from cloud archive w/
:> :xenial-lts kernels (4.4.0)
:> :
:> :-Jon
:> :
:> :-- 
:> 
:> _______________________________________________
:> OpenStack-operators mailing list
:> OpenStack-operators at lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators