I’ve made few test this morning, disabling port security on the MGMT port of VM-A (bastion) indeed did the trick and the traffic started to flow correctly.
I’ve then tested to enable it back and set the allowed_pairs with subnet, it works too, so the issue is definitely what you were pointing out so big shout out to you as this options is so deeply burried on the config/ui that I didn’t thought of it more than that.
Yeah regarding k8s that what I was suspecting, thanks for the confirmation.
However, there is one thing that I can’t get my head around with my simple router installation.
If VM-X send a ping for let say google.com on its gateway (VM-A), the packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward know that this packet is belonging to another nic which is having the appropriate default gareway. At this point, we didn’t touched anything on the packet (to my knowledge) so once we reach the SNAT our packet header source IP is changed with the external router connected nic IP, then it flow through dns resolver that answer this IP, once the nic get this response, it just forward it back to the MGMT nic of the bastion which in turn answer to the VM-X mgmt nic without doing any spoofing etc.
I’m feeling that I miss something on the way in here, what did I missed on this flow? At which step VM-A is breaking the law? At forward step?
Le mer. 9 mars 2022 à 03:20, Laurent Dumont laurentfdumont@gmail.com a écrit :
Any overlay inside the vxlan overlay should work. Since it encapsulates the actual pod/service traffic, Openstack has no clue and happily forwards the traffic since it matches the spec (mac + IP) of the port.
Using only security groups will not let you spoof traffic/act as a router.
- If you want to keep using security groups, you will need
allowed-pairs. - You can use a subnet as a wildcard / or use the IP address specifically.
- If you dont mind very permissive rules, then disabling port-security
will require you to remove security groups + allowed-ip-pairs.
On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND gael.therond@bitswalk.com wrote:
Interesting I’ve thought about especially because of the anti-spoofing features but I had the feeling that Security-groups if assigned to the port would have allowed it.
Additionally, what’s really weird is that one of our k8s clusters using flannel and/or calico (depends on the user needs) is actually putting those exact rules (SNAT+FORWARD rules) automatically for pods services ingress access but I’m wondering if it works because the traffic is on a tunnel that the host can’t see.
I’ll have a look at your idea by disabling the port security and check if it work like that. If it works then I’ll re-enable it and work with the allowed-pairs.
Many thanks!
Le mar. 8 mars 2022 à 22:13, Laurent Dumont laurentfdumont@gmail.com a écrit :
You might want to look at port-security on the bastion host VM. If it's enabled, it means that Openstack will drop any outgoing packets that are not sourced using the ip address + mac AND/OR the allowed_pairs defined on the port.
On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND gael.therond@bitswalk.com wrote:
Hi everyone!
I’m facing a weird situation on a tenant of one of our Openstack cluster based on Victoria.
On this tenant, the network topology is as follow:
One DMZ network (192.168.0.0/24) linked to our public network through a neutron router where there is a VM acting as a bastion/router for the MGMT network.
One MGMT network (172.16.31.0/24) where all VMs are linked to.
On the DMZ network, there is a linux Debian 11, let’s call it VM-A with a Floating IP from the public pool, this VM is both attached to the DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 / 172.16.31.23).
All other VMs, let’s call them VM-X are exclusively attached to the MGMT network (ens4).
I’ve setup VM-A with ip_forward kernel module and the following iptables rule:
# iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source 192.168.0.12
My VM-X are on their own setup with a default gateway via VM-A:
# ip route add default via 172.31.16.23
The setup seems to be working as if I don’t put the iptables rule and the kernel forwarding I can’t see any packets on my DMZ interface (ens3) on VM-A from VM-X.
Ok so now that you get the whole schema, let dive into the issue.
So when all rules, modules and gateway are set, I can fully see my VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A (ens4) then forwarded to VM-A (ens3) and finally going to our public IP targeted service.
What’s not working however is the response not reaching back to VM-X.
I’ve tcpdump the whole traffic from VM-X to VM-A on each point of the platfrom:
from inside the VM-X nic, on the tap device, on the qbr bridge, on the qvb veth, on the qvo second side of the veth through the ovs bridges and vice-versa.
However the response packets aren’t reaching back further than on the VM-A qvo veth. Once it exit the VM-A the traffic never reaches the VM-X.
What’s really suspicious in here is that a direct ping from VM-X (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks like if ovs detected that the response on a SNAT case isn’t legit or something similar.
Is anyone able to get such setup working?
Here are few additional information: Host runs on CentOS 8.5 latest update. Our platform is a Openstack Victoria deployed using kolla-ansible. We are using a OVS based deployment. Our tunnels are VXLAN. All VMs have a fully open secgroup applied and all ports have it (I checked it twice and even on host iptables).
If you ever need additional information feel free to let me know !