Open Stack

I actually dig deeper and that’s exactly that, it’s filtered out by iptable
that get a special match rule when you don’t disable the security ports or
assign any allowed pairs :-)

TBN: Your explication and schema is perfect! I’ll use it to explain this to
my teammates that may face it at some point.

Le ven. 11 mars 2022 à 23:37, Laurent Dumont <laurentfdumont at gmail.com> a
écrit :

> It's a bit hard to see without a diagram but here is my theory.
>
>    - Packets from client-vm towards external endpoint exit from eth0.
>    - Packets arrive to eth1 on router-vm and are allowed because
>       - security-groups OK
>       - mac-address + ip_address policy is only applicable to outgoing
>       traffic (that's what I think for now...)
>    - You have a SNAT in place for eth0.
>    - Packets leave router-vm from eth0 with the source address becoming
>    the IP of eth0. This is allowed because
>       - security-groups OK
>       - mac-address + ip_address policies are applied, but will match the
>       packet since you are SNAT the traffic (and not just routing).
>    - Packets arrive to ze-internet, traffic returns towards the router-vm.
>    - Packets arrive to eth0, still valid.
>    - Packets are processed by iptables which mangles the flow to correct
>    the source IP to the source of the ze-internet.
>    - Packets try to leave through eth1 but are now invalid because
>       - security-groups OK
>       - mac-address + ip_address policies are applied *but are now
>       invalid. Source is ze-internet, which Openstack knows is not associated
>       with eth1.*
>       - If you tcpdump on eth1 of router-vm, you should see the traffic
>    going out since it's before any filtering from ovs on the compute. Either
>    ovs-flows are iptables are used for the actual filtering.
>
>
> At least, that's what I think. I'll run some tests in the lab!
> [image: image.png]
>
> On Wed, Mar 9, 2022 at 4:51 AM Gaël THEROND <gael.therond at bitswalk.com>
> wrote:
>
>> I’ve made few test this morning, disabling port security on the MGMT port
>> of VM-A (bastion) indeed did the trick and the traffic started to flow
>> correctly.
>>
>> I’ve then tested to enable it back and set the allowed_pairs with subnet,
>> it works too, so the issue is definitely what you were pointing out so big
>> shout out to you as this options is so deeply burried on the config/ui that
>> I didn’t thought of it more than that.
>>
>> Yeah regarding k8s that what I was suspecting, thanks for the
>> confirmation.
>>
>> However, there is one thing that I can’t get my head around with my
>> simple router installation.
>>
>> If VM-X send a ping for let say google.com on its gateway (VM-A), the
>> packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward
>> know that this packet is belonging to another nic which is having the
>> appropriate default gareway. At this point, we didn’t touched anything on
>> the packet (to my knowledge) so once we reach the SNAT our packet header
>> source IP is changed with the external router connected nic IP, then it
>> flow through dns resolver that answer this IP, once the nic get this
>> response, it just forward it back to the MGMT nic of the bastion which in
>> turn answer to the VM-X mgmt nic without doing any spoofing etc.
>>
>> I’m feeling that I miss something on the way in here, what did I missed
>> on this flow? At which step VM-A is breaking the law? At forward step?
>>
>> Le mer. 9 mars 2022 à 03:20, Laurent Dumont <laurentfdumont at gmail.com> a
>> écrit :
>>
>>> Any overlay inside the vxlan overlay should work. Since it encapsulates
>>> the actual pod/service traffic, Openstack has no clue and happily forwards
>>> the traffic since it matches the spec (mac + IP) of the port.
>>>
>>> Using only security groups will not let you spoof traffic/act as a
>>> router.
>>>
>>>    - If you want to keep using security groups, you will need
>>>    allowed-pairs.
>>>       - You can use a subnet as a wildcard / or use the IP address
>>>       specifically.
>>>    - If you dont mind very permissive rules, then disabling
>>>    port-security will require you to remove security groups + allowed-ip-pairs.
>>>
>>>
>>> On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND <gael.therond at bitswalk.com>
>>> wrote:
>>>
>>>> Interesting I’ve thought about especially because of the anti-spoofing
>>>> features but I had the feeling that Security-groups if assigned to the port
>>>> would have allowed it.
>>>>
>>>> Additionally, what’s really weird is that one of our k8s clusters using
>>>> flannel and/or calico (depends on the user needs) is actually putting those
>>>> exact rules (SNAT+FORWARD rules) automatically for pods services ingress
>>>> access but I’m wondering if it works because the traffic is on a tunnel
>>>> that the host can’t see.
>>>>
>>>> I’ll have a look at your idea by disabling the port security and check
>>>> if it work like that. If it works then I’ll re-enable it and work with the
>>>> allowed-pairs.
>>>>
>>>> Many thanks!
>>>>
>>>> Le mar. 8 mars 2022 à 22:13, Laurent Dumont <laurentfdumont at gmail.com>
>>>> a écrit :
>>>>
>>>>> You might want to look at port-security on the bastion host VM. If
>>>>> it's enabled, it means that Openstack will drop any outgoing packets that
>>>>> are not sourced using the ip address + mac AND/OR the allowed_pairs defined
>>>>> on the port.
>>>>>
>>>>> On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND <gael.therond at bitswalk.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone!
>>>>>>
>>>>>> I’m facing a weird situation on a tenant of one of our Openstack
>>>>>> cluster based on Victoria.
>>>>>>
>>>>>> On this tenant, the network topology is as follow:
>>>>>>
>>>>>> One DMZ network (192.168.0.0/24) linked to our public network
>>>>>> through a neutron router where there is a VM acting as a bastion/router for
>>>>>> the MGMT network.
>>>>>>
>>>>>> One MGMT network (172.16.31.0/24) where all VMs are linked to.
>>>>>>
>>>>>> On the DMZ network, there is a linux Debian 11, let’s call it VM-A
>>>>>> with a Floating IP from the public pool, this VM is both attached to the
>>>>>> DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 /
>>>>>> 172.16.31.23).
>>>>>>
>>>>>> All other VMs, let’s call them VM-X are exclusively attached to the
>>>>>> MGMT network (ens4).
>>>>>>
>>>>>> I’ve setup VM-A with ip_forward kernel module and the following
>>>>>> iptables rule:
>>>>>>
>>>>>> # iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source
>>>>>> 192.168.0.12
>>>>>>
>>>>>> My VM-X are on their own setup with a default gateway via VM-A:
>>>>>>
>>>>>> # ip route add default via 172.31.16.23
>>>>>>
>>>>>> The setup seems to be working as if I don’t put the iptables rule and
>>>>>> the kernel forwarding I can’t see any packets on my DMZ interface (ens3) on
>>>>>> VM-A from VM-X.
>>>>>>
>>>>>> Ok so now that you get the whole schema, let dive into the issue.
>>>>>>
>>>>>> So when all rules, modules and gateway are set, I can fully see my
>>>>>> VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A
>>>>>> (ens4) then forwarded to VM-A (ens3) and finally going to our public IP
>>>>>> targeted service.
>>>>>>
>>>>>> What’s not working however is the response not reaching back to VM-X.
>>>>>>
>>>>>> I’ve tcpdump the whole traffic from VM-X to VM-A on each point of the
>>>>>> platfrom:
>>>>>>
>>>>>> from inside the VM-X nic, on the tap device, on the qbr bridge, on
>>>>>> the qvb veth, on the qvo second side of the veth through the ovs bridges
>>>>>> and vice-versa.
>>>>>>
>>>>>> However the response packets aren’t reaching back further than on the
>>>>>> VM-A qvo veth.
>>>>>> Once it exit the VM-A the traffic never reaches the VM-X.
>>>>>>
>>>>>> What’s really suspicious in here is that a direct ping from VM-X
>>>>>> (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks
>>>>>> like if ovs detected that the response on a SNAT case isn’t legit or
>>>>>> something similar.
>>>>>>
>>>>>> Is anyone able to get such setup working?
>>>>>>
>>>>>> Here are few additional information:
>>>>>> Host runs on CentOS 8.5 latest update.
>>>>>> Our platform is a Openstack Victoria deployed using kolla-ansible.
>>>>>> We are using a OVS based deployment.
>>>>>> Our tunnels are VXLAN.
>>>>>> All VMs have a fully open secgroup applied and all ports have it (I
>>>>>> checked it twice and even on host iptables).
>>>>>>
>>>>>> If you ever need additional information feel free to let me know !
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220312/85b80ba9/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 21446 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220312/85b80ba9/attachment-0001.png>