Open Stack

Nice! I reread and meant to say that SNAT transforms the packets to take
the source of eth1, not eth0.

But the logic is the same.

It's worth mentioning that routers in Openstack essentially do that. But
they are a bit less flexible than a VM.

On Fri, Mar 11, 2022, 6:06 PM Gaël THEROND <gael.therond at bitswalk.com>
wrote:

> I actually dig deeper and that’s exactly that, it’s filtered out by
> iptable that get a special match rule when you don’t disable the security
> ports or assign any allowed pairs :-)
>
> TBN: Your explication and schema is perfect! I’ll use it to explain this
> to my teammates that may face it at some point.
>
> Le ven. 11 mars 2022 à 23:37, Laurent Dumont <laurentfdumont at gmail.com> a
> écrit :
>
>> It's a bit hard to see without a diagram but here is my theory.
>>
>>    - Packets from client-vm towards external endpoint exit from eth0.
>>    - Packets arrive to eth1 on router-vm and are allowed because
>>       - security-groups OK
>>       - mac-address + ip_address policy is only applicable to outgoing
>>       traffic (that's what I think for now...)
>>    - You have a SNAT in place for eth0.
>>    - Packets leave router-vm from eth0 with the source address becoming
>>    the IP of eth0. This is allowed because
>>       - security-groups OK
>>       - mac-address + ip_address policies are applied, but will match
>>       the packet since you are SNAT the traffic (and not just routing).
>>    - Packets arrive to ze-internet, traffic returns towards the
>>    router-vm.
>>    - Packets arrive to eth0, still valid.
>>    - Packets are processed by iptables which mangles the flow to correct
>>    the source IP to the source of the ze-internet.
>>    - Packets try to leave through eth1 but are now invalid because
>>       - security-groups OK
>>       - mac-address + ip_address policies are applied *but are now
>>       invalid. Source is ze-internet, which Openstack knows is not associated
>>       with eth1.*
>>       - If you tcpdump on eth1 of router-vm, you should see the traffic
>>    going out since it's before any filtering from ovs on the compute. Either
>>    ovs-flows are iptables are used for the actual filtering.
>>
>>
>> At least, that's what I think. I'll run some tests in the lab!
>> [image: image.png]
>>
>> On Wed, Mar 9, 2022 at 4:51 AM Gaël THEROND <gael.therond at bitswalk.com>
>> wrote:
>>
>>> I’ve made few test this morning, disabling port security on the MGMT
>>> port of VM-A (bastion) indeed did the trick and the traffic started to flow
>>> correctly.
>>>
>>> I’ve then tested to enable it back and set the allowed_pairs with
>>> subnet, it works too, so the issue is definitely what you were pointing out
>>> so big shout out to you as this options is so deeply burried on the
>>> config/ui that I didn’t thought of it more than that.
>>>
>>> Yeah regarding k8s that what I was suspecting, thanks for the
>>> confirmation.
>>>
>>> However, there is one thing that I can’t get my head around with my
>>> simple router installation.
>>>
>>> If VM-X send a ping for let say google.com on its gateway (VM-A), the
>>> packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward
>>> know that this packet is belonging to another nic which is having the
>>> appropriate default gareway. At this point, we didn’t touched anything on
>>> the packet (to my knowledge) so once we reach the SNAT our packet header
>>> source IP is changed with the external router connected nic IP, then it
>>> flow through dns resolver that answer this IP, once the nic get this
>>> response, it just forward it back to the MGMT nic of the bastion which in
>>> turn answer to the VM-X mgmt nic without doing any spoofing etc.
>>>
>>> I’m feeling that I miss something on the way in here, what did I missed
>>> on this flow? At which step VM-A is breaking the law? At forward step?
>>>
>>> Le mer. 9 mars 2022 à 03:20, Laurent Dumont <laurentfdumont at gmail.com>
>>> a écrit :
>>>
>>>> Any overlay inside the vxlan overlay should work. Since it encapsulates
>>>> the actual pod/service traffic, Openstack has no clue and happily forwards
>>>> the traffic since it matches the spec (mac + IP) of the port.
>>>>
>>>> Using only security groups will not let you spoof traffic/act as a
>>>> router.
>>>>
>>>>    - If you want to keep using security groups, you will need
>>>>    allowed-pairs.
>>>>       - You can use a subnet as a wildcard / or use the IP address
>>>>       specifically.
>>>>    - If you dont mind very permissive rules, then disabling
>>>>    port-security will require you to remove security groups + allowed-ip-pairs.
>>>>
>>>>
>>>> On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND <gael.therond at bitswalk.com>
>>>> wrote:
>>>>
>>>>> Interesting I’ve thought about especially because of the anti-spoofing
>>>>> features but I had the feeling that Security-groups if assigned to the port
>>>>> would have allowed it.
>>>>>
>>>>> Additionally, what’s really weird is that one of our k8s clusters
>>>>> using flannel and/or calico (depends on the user needs) is actually putting
>>>>> those exact rules (SNAT+FORWARD rules) automatically for pods services
>>>>> ingress access but I’m wondering if it works because the traffic is on a
>>>>> tunnel that the host can’t see.
>>>>>
>>>>> I’ll have a look at your idea by disabling the port security and check
>>>>> if it work like that. If it works then I’ll re-enable it and work with the
>>>>> allowed-pairs.
>>>>>
>>>>> Many thanks!
>>>>>
>>>>> Le mar. 8 mars 2022 à 22:13, Laurent Dumont <laurentfdumont at gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> You might want to look at port-security on the bastion host VM. If
>>>>>> it's enabled, it means that Openstack will drop any outgoing packets that
>>>>>> are not sourced using the ip address + mac AND/OR the allowed_pairs defined
>>>>>> on the port.
>>>>>>
>>>>>> On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND <
>>>>>> gael.therond at bitswalk.com> wrote:
>>>>>>
>>>>>>> Hi everyone!
>>>>>>>
>>>>>>> I’m facing a weird situation on a tenant of one of our Openstack
>>>>>>> cluster based on Victoria.
>>>>>>>
>>>>>>> On this tenant, the network topology is as follow:
>>>>>>>
>>>>>>> One DMZ network (192.168.0.0/24) linked to our public network
>>>>>>> through a neutron router where there is a VM acting as a bastion/router for
>>>>>>> the MGMT network.
>>>>>>>
>>>>>>> One MGMT network (172.16.31.0/24) where all VMs are linked to.
>>>>>>>
>>>>>>> On the DMZ network, there is a linux Debian 11, let’s call it VM-A
>>>>>>> with a Floating IP from the public pool, this VM is both attached to the
>>>>>>> DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 /
>>>>>>> 172.16.31.23).
>>>>>>>
>>>>>>> All other VMs, let’s call them VM-X are exclusively attached to the
>>>>>>> MGMT network (ens4).
>>>>>>>
>>>>>>> I’ve setup VM-A with ip_forward kernel module and the following
>>>>>>> iptables rule:
>>>>>>>
>>>>>>> # iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source
>>>>>>> 192.168.0.12
>>>>>>>
>>>>>>> My VM-X are on their own setup with a default gateway via VM-A:
>>>>>>>
>>>>>>> # ip route add default via 172.31.16.23
>>>>>>>
>>>>>>> The setup seems to be working as if I don’t put the iptables rule
>>>>>>> and the kernel forwarding I can’t see any packets on my DMZ interface
>>>>>>> (ens3) on VM-A from VM-X.
>>>>>>>
>>>>>>> Ok so now that you get the whole schema, let dive into the issue.
>>>>>>>
>>>>>>> So when all rules, modules and gateway are set, I can fully see my
>>>>>>> VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A
>>>>>>> (ens4) then forwarded to VM-A (ens3) and finally going to our public IP
>>>>>>> targeted service.
>>>>>>>
>>>>>>> What’s not working however is the response not reaching back to VM-X.
>>>>>>>
>>>>>>> I’ve tcpdump the whole traffic from VM-X to VM-A on each point of
>>>>>>> the platfrom:
>>>>>>>
>>>>>>> from inside the VM-X nic, on the tap device, on the qbr bridge, on
>>>>>>> the qvb veth, on the qvo second side of the veth through the ovs bridges
>>>>>>> and vice-versa.
>>>>>>>
>>>>>>> However the response packets aren’t reaching back further than on
>>>>>>> the VM-A qvo veth.
>>>>>>> Once it exit the VM-A the traffic never reaches the VM-X.
>>>>>>>
>>>>>>> What’s really suspicious in here is that a direct ping from VM-X
>>>>>>> (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks
>>>>>>> like if ovs detected that the response on a SNAT case isn’t legit or
>>>>>>> something similar.
>>>>>>>
>>>>>>> Is anyone able to get such setup working?
>>>>>>>
>>>>>>> Here are few additional information:
>>>>>>> Host runs on CentOS 8.5 latest update.
>>>>>>> Our platform is a Openstack Victoria deployed using kolla-ansible.
>>>>>>> We are using a OVS based deployment.
>>>>>>> Our tunnels are VXLAN.
>>>>>>> All VMs have a fully open secgroup applied and all ports have it (I
>>>>>>> checked it twice and even on host iptables).
>>>>>>>
>>>>>>> If you ever need additional information feel free to let me know !
>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220313/9495b37e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 21446 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220313/9495b37e/attachment-0001.png>