Open Stack

It's a bit hard to see without a diagram but here is my theory.

   - Packets from client-vm towards external endpoint exit from eth0.
   - Packets arrive to eth1 on router-vm and are allowed because
      - security-groups OK
      - mac-address + ip_address policy is only applicable to outgoing
      traffic (that's what I think for now...)
   - You have a SNAT in place for eth0.
   - Packets leave router-vm from eth0 with the source address becoming the
   IP of eth0. This is allowed because
      - security-groups OK
      - mac-address + ip_address policies are applied, but will match the
      packet since you are SNAT the traffic (and not just routing).
   - Packets arrive to ze-internet, traffic returns towards the router-vm.
   - Packets arrive to eth0, still valid.
   - Packets are processed by iptables which mangles the flow to correct
   the source IP to the source of the ze-internet.
   - Packets try to leave through eth1 but are now invalid because
      - security-groups OK
      - mac-address + ip_address policies are applied *but are now invalid.
      Source is ze-internet, which Openstack knows is not associated with eth1.*
      - If you tcpdump on eth1 of router-vm, you should see the traffic
   going out since it's before any filtering from ovs on the compute. Either
   ovs-flows are iptables are used for the actual filtering.


At least, that's what I think. I'll run some tests in the lab!
[image: image.png]

On Wed, Mar 9, 2022 at 4:51 AM Gaël THEROND <gael.therond at bitswalk.com>
wrote:

> I’ve made few test this morning, disabling port security on the MGMT port
> of VM-A (bastion) indeed did the trick and the traffic started to flow
> correctly.
>
> I’ve then tested to enable it back and set the allowed_pairs with subnet,
> it works too, so the issue is definitely what you were pointing out so big
> shout out to you as this options is so deeply burried on the config/ui that
> I didn’t thought of it more than that.
>
> Yeah regarding k8s that what I was suspecting, thanks for the confirmation.
>
> However, there is one thing that I can’t get my head around with my simple
> router installation.
>
> If VM-X send a ping for let say google.com on its gateway (VM-A), the
> packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward
> know that this packet is belonging to another nic which is having the
> appropriate default gareway. At this point, we didn’t touched anything on
> the packet (to my knowledge) so once we reach the SNAT our packet header
> source IP is changed with the external router connected nic IP, then it
> flow through dns resolver that answer this IP, once the nic get this
> response, it just forward it back to the MGMT nic of the bastion which in
> turn answer to the VM-X mgmt nic without doing any spoofing etc.
>
> I’m feeling that I miss something on the way in here, what did I missed on
> this flow? At which step VM-A is breaking the law? At forward step?
>
> Le mer. 9 mars 2022 à 03:20, Laurent Dumont <laurentfdumont at gmail.com> a
> écrit :
>
>> Any overlay inside the vxlan overlay should work. Since it encapsulates
>> the actual pod/service traffic, Openstack has no clue and happily forwards
>> the traffic since it matches the spec (mac + IP) of the port.
>>
>> Using only security groups will not let you spoof traffic/act as a router.
>>
>>    - If you want to keep using security groups, you will need
>>    allowed-pairs.
>>       - You can use a subnet as a wildcard / or use the IP address
>>       specifically.
>>    - If you dont mind very permissive rules, then disabling
>>    port-security will require you to remove security groups + allowed-ip-pairs.
>>
>>
>> On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND <gael.therond at bitswalk.com>
>> wrote:
>>
>>> Interesting I’ve thought about especially because of the anti-spoofing
>>> features but I had the feeling that Security-groups if assigned to the port
>>> would have allowed it.
>>>
>>> Additionally, what’s really weird is that one of our k8s clusters using
>>> flannel and/or calico (depends on the user needs) is actually putting those
>>> exact rules (SNAT+FORWARD rules) automatically for pods services ingress
>>> access but I’m wondering if it works because the traffic is on a tunnel
>>> that the host can’t see.
>>>
>>> I’ll have a look at your idea by disabling the port security and check
>>> if it work like that. If it works then I’ll re-enable it and work with the
>>> allowed-pairs.
>>>
>>> Many thanks!
>>>
>>> Le mar. 8 mars 2022 à 22:13, Laurent Dumont <laurentfdumont at gmail.com>
>>> a écrit :
>>>
>>>> You might want to look at port-security on the bastion host VM. If it's
>>>> enabled, it means that Openstack will drop any outgoing packets that are
>>>> not sourced using the ip address + mac AND/OR the allowed_pairs defined on
>>>> the port.
>>>>
>>>> On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND <gael.therond at bitswalk.com>
>>>> wrote:
>>>>
>>>>> Hi everyone!
>>>>>
>>>>> I’m facing a weird situation on a tenant of one of our Openstack
>>>>> cluster based on Victoria.
>>>>>
>>>>> On this tenant, the network topology is as follow:
>>>>>
>>>>> One DMZ network (192.168.0.0/24) linked to our public network through
>>>>> a neutron router where there is a VM acting as a bastion/router for the
>>>>> MGMT network.
>>>>>
>>>>> One MGMT network (172.16.31.0/24) where all VMs are linked to.
>>>>>
>>>>> On the DMZ network, there is a linux Debian 11, let’s call it VM-A
>>>>> with a Floating IP from the public pool, this VM is both attached to the
>>>>> DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 /
>>>>> 172.16.31.23).
>>>>>
>>>>> All other VMs, let’s call them VM-X are exclusively attached to the
>>>>> MGMT network (ens4).
>>>>>
>>>>> I’ve setup VM-A with ip_forward kernel module and the following
>>>>> iptables rule:
>>>>>
>>>>> # iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source
>>>>> 192.168.0.12
>>>>>
>>>>> My VM-X are on their own setup with a default gateway via VM-A:
>>>>>
>>>>> # ip route add default via 172.31.16.23
>>>>>
>>>>> The setup seems to be working as if I don’t put the iptables rule and
>>>>> the kernel forwarding I can’t see any packets on my DMZ interface (ens3) on
>>>>> VM-A from VM-X.
>>>>>
>>>>> Ok so now that you get the whole schema, let dive into the issue.
>>>>>
>>>>> So when all rules, modules and gateway are set, I can fully see my
>>>>> VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A
>>>>> (ens4) then forwarded to VM-A (ens3) and finally going to our public IP
>>>>> targeted service.
>>>>>
>>>>> What’s not working however is the response not reaching back to VM-X.
>>>>>
>>>>> I’ve tcpdump the whole traffic from VM-X to VM-A on each point of the
>>>>> platfrom:
>>>>>
>>>>> from inside the VM-X nic, on the tap device, on the qbr bridge, on the
>>>>> qvb veth, on the qvo second side of the veth through the ovs bridges and
>>>>> vice-versa.
>>>>>
>>>>> However the response packets aren’t reaching back further than on the
>>>>> VM-A qvo veth.
>>>>> Once it exit the VM-A the traffic never reaches the VM-X.
>>>>>
>>>>> What’s really suspicious in here is that a direct ping from VM-X
>>>>> (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks
>>>>> like if ovs detected that the response on a SNAT case isn’t legit or
>>>>> something similar.
>>>>>
>>>>> Is anyone able to get such setup working?
>>>>>
>>>>> Here are few additional information:
>>>>> Host runs on CentOS 8.5 latest update.
>>>>> Our platform is a Openstack Victoria deployed using kolla-ansible.
>>>>> We are using a OVS based deployment.
>>>>> Our tunnels are VXLAN.
>>>>> All VMs have a fully open secgroup applied and all ports have it (I
>>>>> checked it twice and even on host iptables).
>>>>>
>>>>> If you ever need additional information feel free to let me know !
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220311/d9c37c62/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 21446 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220311/d9c37c62/attachment-0001.png>