Open Stack

This is a new relatively new issue that has started occurring on our
OpenStack setup since we upgraded Juno (RDO based) to 2.2-1 (we had been
running Juno 2.0 before without issue). Our setup is:

1 x controller
1 x neutron
4 x compute

backed by a 3 node Ceph cluster, running CentOS 7.1

The neutron network is setup with GRE tunnels from the neutron server to
the compute nodes using OVS on a separate network interface from the
management and SAN traffic.

It has taken us a while to figure out what is going on, but basically when
an instance is created and scheduled on a compute node, neutron is applying
iptables rules to all compute nodes. These iptables rules block new
incoming connections, however established sessions continue to work (like
an ongoing ping will continue to work, but if we stop it and restart it, it
then stops working).

We have found by looking at tcpdump that packets are making it all the way
to the qbrxxx interfaces, but then are being dropped before reaching the
tapxxxx interface which seems to mean that iptables rules are the culprit.
If we flush the iptables rules, then communication is restored and
everything works, again seemingly confirming the iptable rules being the
problem.

We initially thought the problem was something to do with Security Groups,
but the issue affects all instances regardless of tenant. It might be an
issue with the default Security Group not being applied even though it is
defined? We have some tenants with custom Security Groups and they are also
affected, so I don't think that is the issue.

I would like to get this resolved, but my iptables-foo is a bit weak in
this regard. Our current workaround is to have a crontab flushing the
tables every minute, but that's down right terrible and we know it. So if
anybody sees anything glaringly obvious as to why this occurring or is
aware of some sort of bug or issue that would explain and maybe address
this issue, I would love to hear about it.

Maybe we are missing something in our iptables rules that should be there.
Perhaps forwarding an example of a working /etc/sysconfig/iptables from
somebody that has a working CentOS 7.1 setup using RDO.

Here are the rules being applied:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
neutron-openvswi-INPUT  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
neutron-filter-top  all  --  0.0.0.0/0            0.0.0.0/0
neutron-openvswi-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
neutron-filter-top  all  --  0.0.0.0/0            0.0.0.0/0
neutron-openvswi-OUTPUT  all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-filter-top (2 references)
target     prot opt source               destination
neutron-openvswi-local  all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-openvswi-FORWARD (1 references)
target     prot opt source               destination
neutron-openvswi-sg-chain  all  --  0.0.0.0/0            0.0.0.0/0
   PHYSDEV match --physdev-out tapf91449ef-0e --physdev-is-bridged
neutron-openvswi-sg-chain  all  --  0.0.0.0/0            0.0.0.0/0
   PHYSDEV match --physdev-in tapf91449ef-0e --physdev-is-bridged

Chain neutron-openvswi-INPUT (1 references)
target     prot opt source               destination
neutron-openvswi-of91449ef-0  all  --  0.0.0.0/0            0.0.0.0/0
     PHYSDEV match --physdev-in tapf91449ef-0e --physdev-is-bridged

Chain neutron-openvswi-OUTPUT (1 references)
target     prot opt source               destination

Chain neutron-openvswi-if91449ef-0 (1 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            state INVALID
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            state
RELATED,ESTABLISHED
RETURN     udp  --  10.10.0.2            0.0.0.0/0            udp spt:67
dpt:68
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            match-set
IPv4c8238a74-1311-4700-9 src
neutron-openvswi-sg-fallback  all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-openvswi-local (1 references)
target     prot opt source               destination

Chain neutron-openvswi-of91449ef-0 (2 references)
target     prot opt source               destination
RETURN     udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:68
dpt:67
neutron-openvswi-sf91449ef-0  all  --  0.0.0.0/0            0.0.0.0/0
DROP       udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:67
dpt:68
DROP       all  --  0.0.0.0/0            0.0.0.0/0            state INVALID
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            state
RELATED,ESTABLISHED
RETURN     icmp --  0.0.0.0/0            0.0.0.0/0
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp multiport
dports 1:24
RETURN     udp  --  0.0.0.0/0            0.0.0.0/0            udp multiport
dports 1:65535
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp multiport
dports 26:65535
neutron-openvswi-sg-fallback  all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-openvswi-sf91449ef-0 (1 references)
target     prot opt source               destination
RETURN     all  --  10.10.0.48           0.0.0.0/0            MAC
FA:16:3E:0B:9D:FB
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-openvswi-sg-chain (2 references)
target     prot opt source               destination
neutron-openvswi-if91449ef-0  all  --  0.0.0.0/0            0.0.0.0/0
     PHYSDEV match --physdev-out tapf91449ef-0e --physdev-is-bridged
neutron-openvswi-of91449ef-0  all  --  0.0.0.0/0            0.0.0.0/0
     PHYSDEV match --physdev-in tapf91449ef-0e --physdev-is-bridged
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain neutron-openvswi-sg-fallback (2 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0


Thanks in advance,

Tom Walsh
ExpressHosting
http://expresshosting.net/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150519/6588c76f/attachment.html>