Open Stack

OpenStack Liberty
Ubuntu 14.04

I have a little strange problem, I’m running a Swift cluster but the proxy nodes reside in a OpenStack tenant. The private network of the tenant is connected to a ha-router on the external storage network.

Now this used to work like a charm, where all my 3 proxy nodes within the tenant were able to connect to the storage network and the ports on each of the Swift nodes, but all of the sudden I lost the connectivity from 2 and now if I spin up new instances within the project I can not connect to the swift nodes, but still I can connect from this only proxy.

I can ping the swift nodes but can not connect to any open ports, [6000/2, 22, etc], here is where it gets a little  I have a none swift node on the network that I can connect to with out problems, the swift nodes are not running a firewall.

root at swift-01:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

The nodes belong to the default security group which has the following rules
Ingress IPv6 Any Any - default Delete Rule
Egress IPv4 Any Any 0.0.0.0/0 - Delete Rule
Egress IPv6 Any Any ::/0 - Delete Rule
Ingress IPv4 Any Any - default Delete Rule
Ingress IPv4 ICMP Any 0.0.0.0/0 - Delete Rule
Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 -

I created a new project and set up a router against the storage network in the same manner as my previous project and instances within that project can connect to ports on all servers running on the storage network.

On one of the network nodes I ran "ip netns exec qrouter-dfa2bdc2-7482-42c4-b166-515849119428 bash” (the router in the faulty project) and tried to ping and telnet to the ports on the swift hosts without luck.

Any ideas on where to go next for troubleshooting ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160929/2d14cc1f/attachment.html>