[ops] In a compute node, outbound traffic blocked when a VM running
Summary of the problem This problem is in a compute node, not in a VM. Once a VM is running in a compute node, all outbound connections in a compute node (not VM) are blocked. For example: # telnet www.google.com 80 Trying 172.217.5.4... Technical Details: We only use provider network. These 4 services are running in each compute node: - neutron-linuxbridge-agent.service - neutron-dhcp-agent.service - neutron-metadata-agent.service - openstack-nova-compute.service Detailed description of the problem: In a compute node, the following is the result of iptables -L when no VM is running: <begin> Chain INPUT (policy ACCEPT) target prot opt source destination neutron-linuxbri-INPUT all -- anywhere anywhere And our usual rules Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-linuxbri-FORWARD all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-linuxbri-OUTPUT all -- anywhere anywhere Chain neutron-filter-top (2 references) target prot opt source destination neutron-linuxbri-local all -- anywhere anywhere Chain neutron-linuxbri-FORWARD (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged Chain neutron-linuxbri-INPUT (1 references) target prot opt source destination Chain neutron-linuxbri-OUTPUT (1 references) target prot opt source destination Chain neutron-linuxbri-local (1 references) target prot opt source destination Chain neutron-linuxbri-sg-chain (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain neutron-linuxbri-sg-fallback (0 references) target prot opt source destination DROP all -- anywhere anywhere </end> In the same compute node, when a VM is running, the following is the result of iptables -L: <begin> Chain INPUT (policy ACCEPT) target prot opt source destination neutron-linuxbri-INPUT all -- anywhere anywhere And our usual rules Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-linuxbri-FORWARD all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-linuxbri-OUTPUT all -- anywhere anywhere Chain neutron-filter-top (2 references) target prot opt source destination neutron-linuxbri-local all -- anywhere anywhere Chain neutron-linuxbri-FORWARD (1 references) target prot opt source destination neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged Chain neutron-linuxbri-INPUT (1 references) target prot opt source destination neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged Chain neutron-linuxbri-OUTPUT (1 references) target prot opt source destination Chain neutron-linuxbri-i8 (1 references) target prot opt source destination RETURN all -- anywhere anywhere state RELATED,ESTABLISHED RETURN udp -- anywhere fermicloud248.fnal.gov udp spt:bootps dpt:bootpc RETURN udp -- anywhere 255.255.255.255 udp spt:bootps dpt:bootpc RETURN icmp -- anywhere anywhere RETURN tcp -- fermilab-net.fnal.gov/16 anywhere tcp dpt:ssh RETURN all -- anywhere anywhere match-set NIPv41d69ba3c-68e3-414f-8f1b- src DROP all -- anywhere anywhere state INVALID neutron-linuxbri-sg-fallback all -- anywhere anywhere Chain neutron-linuxbri-local (1 references) target prot opt source destination Chain neutron-linuxbri-o8 (2 references) target prot opt source destination RETURN udp -- default 255.255.255.255 udp spt:bootpc dpt:bootps neutron-linuxbri-s8 all -- anywhere anywhere RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc RETURN all -- anywhere anywhere state RELATED,ESTABLISHED RETURN tcp -- anywhere anywhere tcp dpt:https RETURN all -- anywhere anywhere RETURN tcp -- anywhere anywhere tcp dpt:http DROP all -- anywhere anywhere state INVALID neutron-linuxbri-sg-fallback all -- anywhere anywhere Chain neutron-linuxbri-s8 (1 references) target prot opt source destination RETURN all -- fermicloud248.fnal.gov anywhere MAC FA:16: DROP all -- anywhere anywhere Chain neutron-linuxbri-sg-chain (2 references) target prot opt source destination neutron-linuxbri-i8 all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged ACCEPT all -- anywhere anywhere Chain neutron-linuxbri-sg-fallback (2 references) target prot opt source destination DROP all -- anywhere anywhere </end> Let me summarize the differences from when no VM running: Chain INPUT : no change Chain FORWARD: no change Chain OUTPUT : no change Chain neutron-filter-top: no change Chain neutron-linuxbri-FORWARD: Two new rules are added neutron-linuxbri-sg-chain neutron-linuxbri-sg-chain Chain neutron-linuxbri-INPUT: One new rule is added neutron-linuxbri-o8ae816b0-f Chain neutron-linuxbri-sg-chain: Two new rules are added neutron-linuxbri-i8 neutron-linuxbri-o8 Chain neutron-linuxbri-OUTPUT: no change Chain neutron-linuxbri-local: no change Chain neutron-linuxbri-sg-fallback: no change Chain neutron-linuxbri-i8: A new chain with multiple rules Chain neutron-linuxbri-o8: A new chain with multiple rules Chain neutron-linuxbri-s8: A new chain with multiple rules But now a problem arises here: All outbound connections are blocked (remember this is in a compute node, not VM): For example: # telnet www.google.com 80 Trying 172.217.5.4... When there isn't any VM running, We don't see this problem. I was wondering if I needed to create a new security group rule for the port 80 (for example) but that didn't solve the issue. Any technical advice will be appreciated, Thanks, Hyunwoo
Hi, On Mon, Dec 07, 2020 at 10:11:19AM -0600, Hyunwoo KIM wrote:
Summary of the problem
This problem is in a compute node, not in a VM.
Once a VM is running in a compute node,
all outbound connections in a compute node (not VM) are blocked.
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
Technical Details:
We only use provider network.
These 4 services are running in each compute node:
- neutron-linuxbridge-agent.service
- neutron-dhcp-agent.service
- neutron-metadata-agent.service
- openstack-nova-compute.service
Detailed description of the problem:
In a compute node, the following is the result of iptables -L when no VM is running:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-sg-chain (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (0 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
In the same compute node, when a VM is running,
the following is the result of iptables -L:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-i8 (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN udp -- anywhere fermicloud248.fnal.gov udp spt:bootps dpt:bootpc
RETURN udp -- anywhere 255.255.255.255 udp spt:bootps dpt:bootpc
RETURN icmp -- anywhere anywhere
RETURN tcp -- fermilab-net.fnal.gov/16 anywhere tcp dpt:ssh
RETURN all -- anywhere anywhere match-set NIPv41d69ba3c-68e3-414f-8f1b- src
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-o8 (2 references)
target prot opt source destination
RETURN udp -- default 255.255.255.255 udp spt:bootpc dpt:bootps
neutron-linuxbri-s8 all -- anywhere anywhere
RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps
DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN tcp -- anywhere anywhere tcp dpt:https
RETURN all -- anywhere anywhere
RETURN tcp -- anywhere anywhere tcp dpt:http
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-s8 (1 references)
target prot opt source destination
RETURN all -- fermicloud248.fnal.gov anywhere MAC FA:16:
DROP all -- anywhere anywhere
Chain neutron-linuxbri-sg-chain (2 references)
target prot opt source destination
neutron-linuxbri-i8 all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
Let me summarize the differences from when no VM running:
Chain INPUT : no change
Chain FORWARD: no change
Chain OUTPUT : no change
Chain neutron-filter-top: no change
Chain neutron-linuxbri-FORWARD: Two new rules are added
neutron-linuxbri-sg-chain
neutron-linuxbri-sg-chain
Chain neutron-linuxbri-INPUT: One new rule is added
neutron-linuxbri-o8ae816b0-f
Chain neutron-linuxbri-sg-chain: Two new rules are added
neutron-linuxbri-i8
neutron-linuxbri-o8
Those are chains which represents rules from Your Security Group used by a VM
Chain neutron-linuxbri-OUTPUT: no change
Chain neutron-linuxbri-local: no change
Chain neutron-linuxbri-sg-fallback: no change
Chain neutron-linuxbri-i8: A new chain with multiple rules
Chain neutron-linuxbri-o8: A new chain with multiple rules
In those 2 chains there are ingress and egress SG rules implemented
Chain neutron-linuxbri-s8: A new chain with multiple rules
And in this one there are antispoofing rules for Your port added.
But now a problem arises here:
All outbound connections are blocked (remember this is in a compute node, not VM):
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
When there isn't any VM running, We don't see this problem.
I was wondering if I needed to create a new security group rule for the port 80 (for example)
but that didn't solve the issue.
Any technical advice will be appreciated,
You should check where exactly Your packets are dropped. Also, You didn't tell us what is the type of the Neutron network to which Your VM is plugged and how bridges are done on Your compute node.
Thanks,
Hyunwoo
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, Thanks for the messages.
You should check where exactly Your packets are dropped. I am communicating with our Linux systems engineers to learn how I can find out which rule is dropping the packets. In the meantime, which method would you recommend for this purpose?
Also, You didn't tell us what is the type of the Neutron network to which Your VM is plugged and The VM is plugged to a provider network.
how bridges are done on Your compute node. We are using Linux Bridges.
Are there any more/other information that I should provide? Thanks, Hyunwoo Application developer at Fermilab. On Mon, Dec 7, 2020 at 3:33 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
On Mon, Dec 07, 2020 at 10:11:19AM -0600, Hyunwoo KIM wrote:
Summary of the problem
This problem is in a compute node, not in a VM.
Once a VM is running in a compute node,
all outbound connections in a compute node (not VM) are blocked.
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
Technical Details:
We only use provider network.
These 4 services are running in each compute node:
- neutron-linuxbridge-agent.service
- neutron-dhcp-agent.service
- neutron-metadata-agent.service
- openstack-nova-compute.service
Detailed description of the problem:
In a compute node, the following is the result of iptables -L when no VM is running:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-sg-chain (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (0 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
In the same compute node, when a VM is running,
the following is the result of iptables -L:
<begin>
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-linuxbri-INPUT all -- anywhere anywhere
And our usual rules
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-linuxbri-OUTPUT all -- anywhere anywhere
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-linuxbri-local all -- anywhere anywhere
Chain neutron-linuxbri-FORWARD (1 references)
target prot opt source destination
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tapb --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap9 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in tap9 --physdev-is-bridged
Chain neutron-linuxbri-INPUT (1 references)
target prot opt source destination
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
Chain neutron-linuxbri-OUTPUT (1 references)
target prot opt source destination
Chain neutron-linuxbri-i8 (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN udp -- anywhere fermicloud248.fnal.gov udp spt:bootps dpt:bootpc
RETURN udp -- anywhere 255.255.255.255 udp spt:bootps dpt:bootpc
RETURN icmp -- anywhere anywhere
RETURN tcp -- fermilab-net.fnal.gov/16 anywhere tcp dpt:ssh
RETURN all -- anywhere anywhere match-set NIPv41d69ba3c-68e3-414f-8f1b- src
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-local (1 references)
target prot opt source destination
Chain neutron-linuxbri-o8 (2 references)
target prot opt source destination
RETURN udp -- default 255.255.255.255 udp spt:bootpc dpt:bootps
neutron-linuxbri-s8 all -- anywhere anywhere
RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps
DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN tcp -- anywhere anywhere tcp dpt:https
RETURN all -- anywhere anywhere
RETURN tcp -- anywhere anywhere tcp dpt:http
DROP all -- anywhere anywhere state INVALID
neutron-linuxbri-sg-fallback all -- anywhere anywhere
Chain neutron-linuxbri-s8 (1 references)
target prot opt source destination
RETURN all -- fermicloud248.fnal.gov anywhere MAC FA:16:
DROP all -- anywhere anywhere
Chain neutron-linuxbri-sg-chain (2 references)
target prot opt source destination
neutron-linuxbri-i8 all -- anywhere anywhere PHYSDEV match --physdev-out tap8 --physdev-is-bridged
neutron-linuxbri-o8 all -- anywhere anywhere PHYSDEV match --physdev-in tap8 --physdev-is-bridged
ACCEPT all -- anywhere anywhere
Chain neutron-linuxbri-sg-fallback (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
</end>
Let me summarize the differences from when no VM running:
Chain INPUT : no change
Chain FORWARD: no change
Chain OUTPUT : no change
Chain neutron-filter-top: no change
Chain neutron-linuxbri-FORWARD: Two new rules are added
neutron-linuxbri-sg-chain
neutron-linuxbri-sg-chain
Chain neutron-linuxbri-INPUT: One new rule is added
neutron-linuxbri-o8ae816b0-f
Chain neutron-linuxbri-sg-chain: Two new rules are added
neutron-linuxbri-i8
neutron-linuxbri-o8
Those are chains which represents rules from Your Security Group used by a VM
Chain neutron-linuxbri-OUTPUT: no change
Chain neutron-linuxbri-local: no change
Chain neutron-linuxbri-sg-fallback: no change
Chain neutron-linuxbri-i8: A new chain with multiple rules
Chain neutron-linuxbri-o8: A new chain with multiple rules
In those 2 chains there are ingress and egress SG rules implemented
Chain neutron-linuxbri-s8: A new chain with multiple rules
And in this one there are antispoofing rules for Your port added.
But now a problem arises here:
All outbound connections are blocked (remember this is in a compute node, not VM):
For example:
# telnet www.google.com 80
Trying 172.217.5.4...
When there isn't any VM running, We don't see this problem.
I was wondering if I needed to create a new security group rule for the port 80 (for example)
but that didn't solve the issue.
Any technical advice will be appreciated,
You should check where exactly Your packets are dropped. Also, You didn't tell us what is the type of the Neutron network to which Your VM is plugged and how bridges are done on Your compute node.
Thanks,
Hyunwoo
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (2)
-
Hyunwoo KIM
-
Slawek Kaplonski