Hi, On Mon, Dec 07, 2020 at 10:11:19AM -0600, Hyunwoo KIM wrote:
Summary of the problem
This problem is in a compute node, not in a VM.
Once a VM is running in a compute node,
all outbound connections in a compute node (not VM) are blocked.
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
Technical Details:
We only use provider network.
These 4 services are running in each compute node:
- neutron-linuxbridge-agent.service
- neutron-dhcp-agent.service
- neutron-metadata-agent.service
- openstack-nova-compute.service
Detailed description of the problem:
In a compute node, the following is the result of iptables -L when no VM is running:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-sg-chain (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (0 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
In the same compute node, when a VM is running,
the following is the result of iptables -L:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-i8 (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN udp -- anywhere fermicloud248.fnal.gov udp spt:bootps dpt:bootpc
RETURN udp -- anywhere 255.255.255.255 udp spt:bootps dpt:bootpc
RETURN icmp -- anywhere anywhere
RETURN tcp -- fermilab-net.fnal.gov/16 anywhere tcp dpt:ssh
RETURN all -- anywhere anywhere match-set NIPv41d69ba3c-68e3-414f-8f1b- src
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-o8 (2 references)
target prot opt source destination
RETURN udp -- default 255.255.255.255 udp spt:bootpc dpt:bootps
neutron-linuxbri-s8 all -- anywhere anywhere
RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps
DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN tcp -- anywhere anywhere tcp dpt:https
RETURN all -- anywhere anywhere
RETURN tcp -- anywhere anywhere tcp dpt:http
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-s8 (1 references)
target prot opt source destination
RETURN all -- fermicloud248.fnal.gov anywhere MAC FA:16:
DROP all -- anywhere anywhere
Chain neutron-linuxbri-sg-chain (2 references)
target prot opt source destination
neutron-linuxbri-i8 all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
Let me summarize the differences from when no VM running:
Chain INPUT : no change
Chain FORWARD: no change
Chain OUTPUT : no change
Chain neutron-filter-top: no change
Chain neutron-linuxbri-FORWARD: Two new rules are added
neutron-linuxbri-sg-chain
neutron-linuxbri-sg-chain
Chain neutron-linuxbri-INPUT: One new rule is added
neutron-linuxbri-o8ae816b0-f
Chain neutron-linuxbri-sg-chain: Two new rules are added
neutron-linuxbri-i8
neutron-linuxbri-o8
Those are chains which represents rules from Your Security Group used by a VM
Chain neutron-linuxbri-OUTPUT: no change
Chain neutron-linuxbri-local: no change
Chain neutron-linuxbri-sg-fallback: no change
Chain neutron-linuxbri-i8: A new chain with multiple rules
Chain neutron-linuxbri-o8: A new chain with multiple rules
In those 2 chains there are ingress and egress SG rules implemented
Chain neutron-linuxbri-s8: A new chain with multiple rules
And in this one there are antispoofing rules for Your port added.
But now a problem arises here:
All outbound connections are blocked (remember this is in a compute node, not VM):
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
When there isn't any VM running, We don't see this problem.
I was wondering if I needed to create a new security group rule for the port 80 (for example)
but that didn't solve the issue.
Any technical advice will be appreciated,
You should check where exactly Your packets are dropped. Also, You didn't tell us what is the type of the Neutron network to which Your VM is plugged and how bridges are done on Your compute node.
Thanks,
Hyunwoo
-- Slawek Kaplonski Principal Software Engineer Red Hat