Open Stack

Yes, if they’re shared and RBAC-ed you can.

I was more about pure private vxlan and a router/function to made them
inter-routable like what I’ve done with the linux base router, but that
would be either suboptimal or complexe depending on the solution chosen as
it would either require:

* a router for vxlans that would be located on your network node but then
it circumvent the purpose of a meshed vxlan topology.

Or

* Complexe as it would involve ovs flow rules and iptables rules on each
compute nodes that host the VMs and that automatically and seamlessly route
your traffic.

Anyway, a linux double homed instance is still the most efficient way (that
I’m aware of) to do that so far.

Le lun. 14 mars 2022 à 23:17, Laurent Dumont <laurentfdumont at gmail.com> a
écrit :

> In theory, if vxlan networks are created with the --shared flag, or
> rbac-ed post creation, you could have one qrouter with multiple vxlan
> ports. You just have to make sure they are not overlapping but I think it
> would route between each out of the box.
>
> On Sun, Mar 13, 2022 at 5:18 PM Gaël THEROND <gael.therond at bitswalk.com>
> wrote:
>
>> Yeah, especially that you can’t use them for internal vxlan to vxlan
>> routing straight from horizon or without having higher knowledge to neutron
>> and using CLI, that honestly my users don’t have :-) I’m working on a
>> ansible skeleton that our business units devs could use « easily » for the
>> less infrastructure oriented.
>>
>> I really think that vxlan to vxlan routing should be allowed right from
>> within horizon for the final users (non ops/admins).
>>
>> Anyway! Thanks a lot!
>>
>> Le dim. 13 mars 2022 à 22:09, Laurent Dumont <laurentfdumont at gmail.com>
>> a écrit :
>>
>>> Nice! I reread and meant to say that SNAT transforms the packets to take
>>> the source of eth1, not eth0.
>>>
>>> But the logic is the same.
>>>
>>> It's worth mentioning that routers in Openstack essentially do that. But
>>> they are a bit less flexible than a VM.
>>>
>>> On Fri, Mar 11, 2022, 6:06 PM Gaël THEROND <gael.therond at bitswalk.com>
>>> wrote:
>>>
>>>> I actually dig deeper and that’s exactly that, it’s filtered out by
>>>> iptable that get a special match rule when you don’t disable the security
>>>> ports or assign any allowed pairs :-)
>>>>
>>>> TBN: Your explication and schema is perfect! I’ll use it to explain
>>>> this to my teammates that may face it at some point.
>>>>
>>>> Le ven. 11 mars 2022 à 23:37, Laurent Dumont <laurentfdumont at gmail.com>
>>>> a écrit :
>>>>
>>>>> It's a bit hard to see without a diagram but here is my theory.
>>>>>
>>>>>    - Packets from client-vm towards external endpoint exit from eth0.
>>>>>    - Packets arrive to eth1 on router-vm and are allowed because
>>>>>       - security-groups OK
>>>>>       - mac-address + ip_address policy is only applicable to
>>>>>       outgoing traffic (that's what I think for now...)
>>>>>    - You have a SNAT in place for eth0.
>>>>>    - Packets leave router-vm from eth0 with the source address
>>>>>    becoming the IP of eth0. This is allowed because
>>>>>       - security-groups OK
>>>>>       - mac-address + ip_address policies are applied, but will match
>>>>>       the packet since you are SNAT the traffic (and not just routing).
>>>>>    - Packets arrive to ze-internet, traffic returns towards the
>>>>>    router-vm.
>>>>>    - Packets arrive to eth0, still valid.
>>>>>    - Packets are processed by iptables which mangles the flow to
>>>>>    correct the source IP to the source of the ze-internet.
>>>>>    - Packets try to leave through eth1 but are now invalid because
>>>>>       - security-groups OK
>>>>>       - mac-address + ip_address policies are applied *but are now
>>>>>       invalid. Source is ze-internet, which Openstack knows is not associated
>>>>>       with eth1.*
>>>>>       - If you tcpdump on eth1 of router-vm, you should see the
>>>>>    traffic going out since it's before any filtering from ovs on the compute.
>>>>>    Either ovs-flows are iptables are used for the actual filtering.
>>>>>
>>>>>
>>>>> At least, that's what I think. I'll run some tests in the lab!
>>>>> [image: image.png]
>>>>>
>>>>> On Wed, Mar 9, 2022 at 4:51 AM Gaël THEROND <gael.therond at bitswalk.com>
>>>>> wrote:
>>>>>
>>>>>> I’ve made few test this morning, disabling port security on the MGMT
>>>>>> port of VM-A (bastion) indeed did the trick and the traffic started to flow
>>>>>> correctly.
>>>>>>
>>>>>> I’ve then tested to enable it back and set the allowed_pairs with
>>>>>> subnet, it works too, so the issue is definitely what you were pointing out
>>>>>> so big shout out to you as this options is so deeply burried on the
>>>>>> config/ui that I didn’t thought of it more than that.
>>>>>>
>>>>>> Yeah regarding k8s that what I was suspecting, thanks for the
>>>>>> confirmation.
>>>>>>
>>>>>> However, there is one thing that I can’t get my head around with my
>>>>>> simple router installation.
>>>>>>
>>>>>> If VM-X send a ping for let say google.com on its gateway (VM-A),
>>>>>> the packet goes with VM-X mac/ip header, then VM-A thanks to the ip_forward
>>>>>> know that this packet is belonging to another nic which is having the
>>>>>> appropriate default gareway. At this point, we didn’t touched anything on
>>>>>> the packet (to my knowledge) so once we reach the SNAT our packet header
>>>>>> source IP is changed with the external router connected nic IP, then it
>>>>>> flow through dns resolver that answer this IP, once the nic get this
>>>>>> response, it just forward it back to the MGMT nic of the bastion which in
>>>>>> turn answer to the VM-X mgmt nic without doing any spoofing etc.
>>>>>>
>>>>>> I’m feeling that I miss something on the way in here, what did I
>>>>>> missed on this flow? At which step VM-A is breaking the law? At forward
>>>>>> step?
>>>>>>
>>>>>> Le mer. 9 mars 2022 à 03:20, Laurent Dumont <laurentfdumont at gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>>> Any overlay inside the vxlan overlay should work. Since it
>>>>>>> encapsulates the actual pod/service traffic, Openstack has no clue and
>>>>>>> happily forwards the traffic since it matches the spec (mac + IP) of the
>>>>>>> port.
>>>>>>>
>>>>>>> Using only security groups will not let you spoof traffic/act as a
>>>>>>> router.
>>>>>>>
>>>>>>>    - If you want to keep using security groups, you will need
>>>>>>>    allowed-pairs.
>>>>>>>       - You can use a subnet as a wildcard / or use the IP address
>>>>>>>       specifically.
>>>>>>>    - If you dont mind very permissive rules, then disabling
>>>>>>>    port-security will require you to remove security groups + allowed-ip-pairs.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 8, 2022 at 4:59 PM Gaël THEROND <
>>>>>>> gael.therond at bitswalk.com> wrote:
>>>>>>>
>>>>>>>> Interesting I’ve thought about especially because of the
>>>>>>>> anti-spoofing features but I had the feeling that Security-groups if
>>>>>>>> assigned to the port would have allowed it.
>>>>>>>>
>>>>>>>> Additionally, what’s really weird is that one of our k8s clusters
>>>>>>>> using flannel and/or calico (depends on the user needs) is actually putting
>>>>>>>> those exact rules (SNAT+FORWARD rules) automatically for pods services
>>>>>>>> ingress access but I’m wondering if it works because the traffic is on a
>>>>>>>> tunnel that the host can’t see.
>>>>>>>>
>>>>>>>> I’ll have a look at your idea by disabling the port security and
>>>>>>>> check if it work like that. If it works then I’ll re-enable it and work
>>>>>>>> with the allowed-pairs.
>>>>>>>>
>>>>>>>> Many thanks!
>>>>>>>>
>>>>>>>> Le mar. 8 mars 2022 à 22:13, Laurent Dumont <
>>>>>>>> laurentfdumont at gmail.com> a écrit :
>>>>>>>>
>>>>>>>>> You might want to look at port-security on the bastion host VM. If
>>>>>>>>> it's enabled, it means that Openstack will drop any outgoing packets that
>>>>>>>>> are not sourced using the ip address + mac AND/OR the allowed_pairs defined
>>>>>>>>> on the port.
>>>>>>>>>
>>>>>>>>> On Tue, Mar 8, 2022 at 3:07 PM Gaël THEROND <
>>>>>>>>> gael.therond at bitswalk.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone!
>>>>>>>>>>
>>>>>>>>>> I’m facing a weird situation on a tenant of one of our Openstack
>>>>>>>>>> cluster based on Victoria.
>>>>>>>>>>
>>>>>>>>>> On this tenant, the network topology is as follow:
>>>>>>>>>>
>>>>>>>>>> One DMZ network (192.168.0.0/24) linked to our public network
>>>>>>>>>> through a neutron router where there is a VM acting as a bastion/router for
>>>>>>>>>> the MGMT network.
>>>>>>>>>>
>>>>>>>>>> One MGMT network (172.16.31.0/24) where all VMs are linked to.
>>>>>>>>>>
>>>>>>>>>> On the DMZ network, there is a linux Debian 11, let’s call it
>>>>>>>>>> VM-A with a Floating IP from the public pool, this VM is both attached to
>>>>>>>>>> the DMZ network (ens3 / 192.168.0.12) AND the MGMT network (ens4 /
>>>>>>>>>> 172.16.31.23).
>>>>>>>>>>
>>>>>>>>>> All other VMs, let’s call them VM-X are exclusively attached to
>>>>>>>>>> the MGMT network (ens4).
>>>>>>>>>>
>>>>>>>>>> I’ve setup VM-A with ip_forward kernel module and the following
>>>>>>>>>> iptables rule:
>>>>>>>>>>
>>>>>>>>>> # iptables -t nat -A POSTROUTING -o ens3 -J SNAT —to-source
>>>>>>>>>> 192.168.0.12
>>>>>>>>>>
>>>>>>>>>> My VM-X are on their own setup with a default gateway via VM-A:
>>>>>>>>>>
>>>>>>>>>> # ip route add default via 172.31.16.23
>>>>>>>>>>
>>>>>>>>>> The setup seems to be working as if I don’t put the iptables rule
>>>>>>>>>> and the kernel forwarding I can’t see any packets on my DMZ interface
>>>>>>>>>> (ens3) on VM-A from VM-X.
>>>>>>>>>>
>>>>>>>>>> Ok so now that you get the whole schema, let dive into the issue.
>>>>>>>>>>
>>>>>>>>>> So when all rules, modules and gateway are set, I can fully see
>>>>>>>>>> my VM-X traffic (ICMP ping to a dns server) going from VM-X (ens4) to VM-A
>>>>>>>>>> (ens4) then forwarded to VM-A (ens3) and finally going to our public IP
>>>>>>>>>> targeted service.
>>>>>>>>>>
>>>>>>>>>> What’s not working however is the response not reaching back to
>>>>>>>>>> VM-X.
>>>>>>>>>>
>>>>>>>>>> I’ve tcpdump the whole traffic from VM-X to VM-A on each point of
>>>>>>>>>> the platfrom:
>>>>>>>>>>
>>>>>>>>>> from inside the VM-X nic, on the tap device, on the qbr bridge,
>>>>>>>>>> on the qvb veth, on the qvo second side of the veth through the ovs bridges
>>>>>>>>>> and vice-versa.
>>>>>>>>>>
>>>>>>>>>> However the response packets aren’t reaching back further than on
>>>>>>>>>> the VM-A qvo veth.
>>>>>>>>>> Once it exit the VM-A the traffic never reaches the VM-X.
>>>>>>>>>>
>>>>>>>>>> What’s really suspicious in here is that a direct ping from VM-X
>>>>>>>>>> (172.16.31.54) to VM-A (172.16.31.23) is coming back correctly, so it looks
>>>>>>>>>> like if ovs detected that the response on a SNAT case isn’t legit or
>>>>>>>>>> something similar.
>>>>>>>>>>
>>>>>>>>>> Is anyone able to get such setup working?
>>>>>>>>>>
>>>>>>>>>> Here are few additional information:
>>>>>>>>>> Host runs on CentOS 8.5 latest update.
>>>>>>>>>> Our platform is a Openstack Victoria deployed using kolla-ansible.
>>>>>>>>>> We are using a OVS based deployment.
>>>>>>>>>> Our tunnels are VXLAN.
>>>>>>>>>> All VMs have a fully open secgroup applied and all ports have it
>>>>>>>>>> (I checked it twice and even on host iptables).
>>>>>>>>>>
>>>>>>>>>> If you ever need additional information feel free to let me know !
>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220315/78e50ac6/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 21446 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220315/78e50ac6/attachment-0001.png>