Open Stack

In theory, if vxlan networks are created with the --shared flag, or rbac-ed
post creation, you could have one qrouter with multiple vxlan ports. You
just have to make sure they are not overlapping but I think it would route
between each out of the box.

On Sun, Mar 13, 2022 at 5:18 PM Gaël THEROND <gael.therond at bitswalk.com>
wrote:

> Yeah, especially that you can’t use them for internal vxlan to vxlan
> routing straight from horizon or without having higher knowledge to neutron
> and using CLI, that honestly my users don’t have :-) I’m working on a
> ansible skeleton that our business units devs could use « easily » for the
> less infrastructure oriented.
>
> I really think that vxlan to vxlan routing should be allowed right from
> within horizon for the final users (non ops/admins).
>
> Anyway! Thanks a lot!
>
> Le dim. 13 mars 2022 à 22:09, Laurent Dumont <laurentfdumont at gmail.com> a
> écrit :
>
>> Nice! I reread and meant to say that SNAT transforms the packets to take
>> the source of eth1, not eth0.
>>
>> But the logic is the same.
>>
>> It's worth mentioning that routers in Openstack essentially do that. But
>> they are a bit less flexible than a VM.
>>
>> On Fri, Mar 11, 2022, 6:06 PM Gaël THEROND <gael.therond at bitswalk.com>
>> wrote:
>>
>>> I actually dig deeper and that’s exactly that, it’s filtered out by
>>> iptable that get a special match rule when you don’t disable the security
>>> ports or assign any allowed pairs :-)
>>>
>>> TBN: Your explication and schema is perfect! I’ll use it to explain this
>>> to my teammates that may face it at some point.
>>>
>>> Le ven. 11 mars 2022 à 23:37, Laurent Dumont <laurentfdumont at gmail.com>
>>> a écrit :
>>>
>>>> It's a bit hard to see without a diagram but here is my theory.
>>>>
>>>>    - Packets from client-vm towards external endpoint exit from eth0.
>>>>    - Packets arrive to eth1 on router-vm and are allowed because
>>>>       - security-groups OK
>>>>       - mac-address + ip_address policy is only applicable to outgoing
>>>>       traffic (that's what I think for now...)
>>>>    - You have a SNAT in place for eth0.
>>>>    - Packets leave router-vm from eth0 with the source address
>>>>    becoming the IP of eth0. This is allowed because
>>>>       - security-groups OK
>>>>       - mac-address + ip_address policies are applied, but will match
>>>>       the packet since you are SNAT the traffic (and not just routing).
>>>>    - Packets arrive to ze-internet, traffic returns towards the
>>>>    router-vm.
>>>>    - Packets arrive to eth0, still valid.
>>>>    - Packets are processed by iptables which mangles the flow to
>>>>    correct the source IP to the source of the ze-internet.
>>>>    - Packets try to leave through eth1 but are now invalid because
>>>>       - security-groups OK
>>>>       - mac-address + ip_address policies are applied *but are now
>>>>       invalid. Source is ze-internet, which Openstack knows is not associated
>>>>       with eth1.*
>>>>       - If you tcpdump on eth1 of router-vm, you should see the
>>>>    traffic going out since it's before any filtering from ovs on the compute.
>>>>    Either ovs-flows are iptables are used for the actual filtering.
>>>>
>>>>
>>>> At least, that's what I think. I'll run some tests in the lab!
>>>> [image: image.png]
>>>>
>>>> On Wed, Mar 9, 2022 at 4:51 AM Gaël THEROND <gael.therond at bitswalk.com>
>>>> wrote:
>>>>
>>>>> I’ve made few test this morning, disabling port security on the MGMT
>>>>> port of VM-A (bastion) indeed did the trick and the traffic started to flow
>>>>> correctly.
>>>>>
>>>>> I’ve then tested to enable it back and set the allowed_pairs with
>>>>> subnet, it works too, so the issue is definitely what you were pointing out
>>>>> so big shout out to you as this options is so deeply burried on the
>>>>> config/ui that I didn’t thought of it more than that.
>>>>>
>>>>> Yeah regarding k8s that what I was suspecting, thanks for the
>>>>> confirmation.
>>>>>
>>>>> However, there is one thing that I can’t get my head around with my
>>>>> simple router installation.
>>>>>
>>>>> If VM-X send a ping for let say google.com on its gateway (VM-A), the
>>>>> packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward
>>>>> know that this packet is belonging to another nic which is having the
>>>>> appropriate default gareway. At this point, we didn’t touched anything on
>>>>> the packet (to my knowledge) so once we reach the SNAT our packet header
>>>>> source IP is changed with the external router connected nic IP, then it
>>>>> flow through dns resolver that answer this IP, once the nic get this
>>>>> response, it just forward it back to the MGMT nic of the bastion which in
>>>>> turn answer to the VM-X mgmt nic without doing any spoofing etc.
>>>>>
>>>>> I’m feeling that I miss something on the way in here, what did I
>>>>> missed on this flow? At which step VM-A is breaking the law? At forward
>>>>> step?
>>>>>
>>>>> Le mer. 9 mars 2022 à 03:20, Laurent Dumont <laurentfdumont at gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Any overlay inside the vxlan overlay should work. Since it
>>>>>> encapsulates the actual pod/service traffic, Openstack has no clue and
>>>>>> happily forwards the traffic since it matches the spec (mac + IP) of the
>>>>>> port.
>>>>>>
>>>>>> Using only security groups will not let you spoof traffic/act as a
>>>>>> router.
>>>>>>
>>>>>>    - If you want to keep using security groups, you will need
>>>>>>    allowed-pairs.
>>>>>>       - You can use a subnet as a wildcard / or use the IP address
>>>>>>       specifically.
>>>>>>    - If you dont mind very permissive rules, then disabling
>>>>>>    port-security will require you to remove security groups + allowed-ip-pairs.
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND <
>>>>>> gael.therond at bitswalk.com> wrote:
>>>>>>
>>>>>>> Interesting I’ve thought about especially because of the
>>>>>>> anti-spoofing features but I had the feeling that Security-groups if
>>>>>>> assigned to the port would have allowed it.
>>>>>>>
>>>>>>> Additionally, what’s really weird is that one of our k8s clusters
>>>>>>> using flannel and/or calico (depends on the user needs) is actually putting
>>>>>>> those exact rules (SNAT+FORWARD rules) automatically for pods services
>>>>>>> ingress access but I’m wondering if it works because the traffic is on a
>>>>>>> tunnel that the host can’t see.
>>>>>>>
>>>>>>> I’ll have a look at your idea by disabling the port security and
>>>>>>> check if it work like that. If it works then I’ll re-enable it and work
>>>>>>> with the allowed-pairs.
>>>>>>>
>>>>>>> Many thanks!
>>>>>>>
>>>>>>> Le mar. 8 mars 2022 à 22:13, Laurent Dumont <
>>>>>>> laurentfdumont at gmail.com> a écrit :
>>>>>>>
>>>>>>>> You might want to look at port-security on the bastion host VM. If
>>>>>>>> it's enabled, it means that Openstack will drop any outgoing packets that
>>>>>>>> are not sourced using the ip address + mac AND/OR the allowed_pairs defined
>>>>>>>> on the port.
>>>>>>>>
>>>>>>>> On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND <
>>>>>>>> gael.therond at bitswalk.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone!
>>>>>>>>>
>>>>>>>>> I’m facing a weird situation on a tenant of one of our Openstack
>>>>>>>>> cluster based on Victoria.
>>>>>>>>>
>>>>>>>>> On this tenant, the network topology is as follow:
>>>>>>>>>
>>>>>>>>> One DMZ network (192.168.0.0/24) linked to our public network
>>>>>>>>> through a neutron router where there is a VM acting as a bastion/router for
>>>>>>>>> the MGMT network.
>>>>>>>>>
>>>>>>>>> One MGMT network (172.16.31.0/24) where all VMs are linked to.
>>>>>>>>>
>>>>>>>>> On the DMZ network, there is a linux Debian 11, let’s call it VM-A
>>>>>>>>> with a Floating IP from the public pool, this VM is both attached to the
>>>>>>>>> DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 /
>>>>>>>>> 172.16.31.23).
>>>>>>>>>
>>>>>>>>> All other VMs, let’s call them VM-X are exclusively attached to
>>>>>>>>> the MGMT network (ens4).
>>>>>>>>>
>>>>>>>>> I’ve setup VM-A with ip_forward kernel module and the following
>>>>>>>>> iptables rule:
>>>>>>>>>
>>>>>>>>> # iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source
>>>>>>>>> 192.168.0.12
>>>>>>>>>
>>>>>>>>> My VM-X are on their own setup with a default gateway via VM-A:
>>>>>>>>>
>>>>>>>>> # ip route add default via 172.31.16.23
>>>>>>>>>
>>>>>>>>> The setup seems to be working as if I don’t put the iptables rule
>>>>>>>>> and the kernel forwarding I can’t see any packets on my DMZ interface
>>>>>>>>> (ens3) on VM-A from VM-X.
>>>>>>>>>
>>>>>>>>> Ok so now that you get the whole schema, let dive into the issue.
>>>>>>>>>
>>>>>>>>> So when all rules, modules and gateway are set, I can fully see my
>>>>>>>>> VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A
>>>>>>>>> (ens4) then forwarded to VM-A (ens3) and finally going to our public IP
>>>>>>>>> targeted service.
>>>>>>>>>
>>>>>>>>> What’s not working however is the response not reaching back to
>>>>>>>>> VM-X.
>>>>>>>>>
>>>>>>>>> I’ve tcpdump the whole traffic from VM-X to VM-A on each point of
>>>>>>>>> the platfrom:
>>>>>>>>>
>>>>>>>>> from inside the VM-X nic, on the tap device, on the qbr bridge, on
>>>>>>>>> the qvb veth, on the qvo second side of the veth through the ovs bridges
>>>>>>>>> and vice-versa.
>>>>>>>>>
>>>>>>>>> However the response packets aren’t reaching back further than on
>>>>>>>>> the VM-A qvo veth.
>>>>>>>>> Once it exit the VM-A the traffic never reaches the VM-X.
>>>>>>>>>
>>>>>>>>> What’s really suspicious in here is that a direct ping from VM-X
>>>>>>>>> (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks
>>>>>>>>> like if ovs detected that the response on a SNAT case isn’t legit or
>>>>>>>>> something similar.
>>>>>>>>>
>>>>>>>>> Is anyone able to get such setup working?
>>>>>>>>>
>>>>>>>>> Here are few additional information:
>>>>>>>>> Host runs on CentOS 8.5 latest update.
>>>>>>>>> Our platform is a Openstack Victoria deployed using kolla-ansible.
>>>>>>>>> We are using a OVS based deployment.
>>>>>>>>> Our tunnels are VXLAN.
>>>>>>>>> All VMs have a fully open secgroup applied and all ports have it
>>>>>>>>> (I checked it twice and even on host iptables).
>>>>>>>>>
>>>>>>>>> If you ever need additional information feel free to let me know !
>>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220314/b0bacf7b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 21446 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220314/b0bacf7b/attachment-0001.png>