[Neutron][OVN] Flat routed provider network has external traffic broadcasted to all VMs
Dear all,
I'm currently debugging a (testing) deployment with neutron/OVN, currently most ports in external flat network. So far I'm happy everything can communicate, but we see a strong performance degradation with increasing number of VMs/ports hosted on a hypervisor. The setup is similar to the one described in the routed provider networks docs [1]. For routing between the different segments, we have physical routers, that are not managed by neutron.
Debugging the issue showed traffic originating from the DC router to VMs go perfectly as unicast traffic through OVN. Communicating from the VM to other networks via the DC router, however, will broadcast the traffic to all attached ports on the internal bridge.
Is there a way to tell neutron/OVN to use a specific MAC address for the router and forward the traffic through the patch port instead of broadcasting it? I could not find any documentation of whether I am missing a neutron port or some other configuration on the hypervisor.
Just to make sure: The VM and DC routers have the right ARP information of itself, the traffic that is duplicated is from VM to router/external network. (via default route to router) As far as I could see, this issue does not appear with private networks, since all MAC addresses are known to OVN/Neutron.
Versions: Nova/Neutron: Yoga ovs-vswitchd (Open vSwitch) 3.1.4 ovn-northd 22.12.1 kernel 5.14.0-284.30.1.el9_2.x86_64
Best, Daniel
[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
Hello Daniel:
* I'm not very good with acronyms: what is DC router? Datacenter I guess, right? * How are you communicating to external resources? Are you using an OVN router and an external network?
Regards.
On Thu, Feb 8, 2024 at 10:28 AM Daniel Failing daniel.failing@cern.ch wrote:
Dear all,
I'm currently debugging a (testing) deployment with neutron/OVN, currently most ports in external flat network. So far I'm happy everything can communicate, but we see a strong performance degradation with increasing number of VMs/ports hosted on a hypervisor. The setup is similar to the one described in the routed provider networks docs [1]. For routing between the different segments, we have physical routers, that are not managed by neutron.
Debugging the issue showed traffic originating from the DC router to VMs go perfectly as unicast traffic through OVN. Communicating from the VM to other networks via the DC router, however, will broadcast the traffic to all attached ports on the internal bridge.
Is there a way to tell neutron/OVN to use a specific MAC address for the router and forward the traffic through the patch port instead of broadcasting it? I could not find any documentation of whether I am missing a neutron port or some other configuration on the hypervisor.
Just to make sure: The VM and DC routers have the right ARP information of itself, the traffic that is duplicated is from VM to router/external network. (via default route to router) As far as I could see, this issue does not appear with private networks, since all MAC addresses are known to OVN/Neutron.
Versions: Nova/Neutron: Yoga ovs-vswitchd (Open vSwitch) 3.1.4 ovn-northd 22.12.1 kernel 5.14.0-284.30.1.el9_2.x86_64
Best, Daniel
[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
Hello Rodolfo,
For DC router, yes, I mean datacenter router, in this case not managed by neutron. We are communicating to external resources through said router. It looks like this: {HV1, HV2, HV3} - switch - router (- end user device, e.g. laptop1). In the hypervisor/ovs: {tap,...} - br-int - patchport - br-eth0 - eth0 There is no OVN router involved in the external network.
The hypervisors and the router are in the same subnet, which is configured in neutron as one subnet. The laptop would be outside of neutron scope, while the router IP is configured as the gateway IP of the subnet.
Best, Daniel ________________________________ From: Rodolfo Alonso Hernandez ralonsoh@redhat.com Sent: Thursday, February 8, 2024 16:24 To: Daniel Failing daniel.failing@cern.ch Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [Neutron][OVN] Flat routed provider network has external traffic broadcasted to all VMs
Hello Daniel:
* I'm not very good with acronyms: what is DC router? Datacenter I guess, right? * How are you communicating to external resources? Are you using an OVN router and an external network?
Regards.
On Thu, Feb 8, 2024 at 10:28 AM Daniel Failing <daniel.failing@cern.chmailto:daniel.failing@cern.ch> wrote: Dear all,
I'm currently debugging a (testing) deployment with neutron/OVN, currently most ports in external flat network. So far I'm happy everything can communicate, but we see a strong performance degradation with increasing number of VMs/ports hosted on a hypervisor. The setup is similar to the one described in the routed provider networks docs [1]. For routing between the different segments, we have physical routers, that are not managed by neutron.
Debugging the issue showed traffic originating from the DC router to VMs go perfectly as unicast traffic through OVN. Communicating from the VM to other networks via the DC router, however, will broadcast the traffic to all attached ports on the internal bridge.
Is there a way to tell neutron/OVN to use a specific MAC address for the router and forward the traffic through the patch port instead of broadcasting it? I could not find any documentation of whether I am missing a neutron port or some other configuration on the hypervisor.
Just to make sure: The VM and DC routers have the right ARP information of itself, the traffic that is duplicated is from VM to router/external network. (via default route to router) As far as I could see, this issue does not appear with private networks, since all MAC addresses are known to OVN/Neutron.
Versions: Nova/Neutron: Yoga ovs-vswitchd (Open vSwitch) 3.1.4 ovn-northd 22.12.1 kernel 5.14.0-284.30.1.el9_2.x86_64
Best, Daniel
[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
Hello Daniel:
OVN uses GW ports (on GW chassis) to egress traffic outside the OVN network. If you are using a flat/vlan network, your configuration will work but only because both the VM (that has a port bound to the OVN mech driver and thus stored in the OVN SB database) and the laptop are in the same broadcast domain. OVN is broadcasting the VM traffic because OVN doesn't know what the laptop IP is.
This configuration is not supported in OpenStack: any access to an external IP should be done through a GW port.
Regards.
On Thu, Feb 8, 2024 at 4:58 PM Daniel Failing daniel.failing@cern.ch wrote:
Hello Rodolfo,
For DC router, yes, I mean datacenter router, in this case not managed by neutron. We are communicating to external resources through said router. It looks like this: {HV1, HV2, HV3} - switch - router (- end user device, e.g. laptop1). In the hypervisor/ovs: {tap,...} - br-int - patchport - br-eth0 - eth0 There is no OVN router involved in the external network.
The hypervisors and the router are in the same subnet, which is configured in neutron as one subnet. The laptop would be outside of neutron scope, while the router IP is configured as the gateway IP of the subnet.
Best, Daniel
*From:* Rodolfo Alonso Hernandez ralonsoh@redhat.com *Sent:* Thursday, February 8, 2024 16:24 *To:* Daniel Failing daniel.failing@cern.ch *Cc:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [Neutron][OVN] Flat routed provider network has external traffic broadcasted to all VMs
Hello Daniel:
- I'm not very good with acronyms: what is DC router? Datacenter I guess,
right?
- How are you communicating to external resources? Are you using an OVN
router and an external network?
Regards.
On Thu, Feb 8, 2024 at 10:28 AM Daniel Failing daniel.failing@cern.ch wrote:
Dear all,
I'm currently debugging a (testing) deployment with neutron/OVN, currently most ports in external flat network. So far I'm happy everything can communicate, but we see a strong performance degradation with increasing number of VMs/ports hosted on a hypervisor. The setup is similar to the one described in the routed provider networks docs [1]. For routing between the different segments, we have physical routers, that are not managed by neutron.
Debugging the issue showed traffic originating from the DC router to VMs go perfectly as unicast traffic through OVN. Communicating from the VM to other networks via the DC router, however, will broadcast the traffic to all attached ports on the internal bridge.
Is there a way to tell neutron/OVN to use a specific MAC address for the router and forward the traffic through the patch port instead of broadcasting it? I could not find any documentation of whether I am missing a neutron port or some other configuration on the hypervisor.
Just to make sure: The VM and DC routers have the right ARP information of itself, the traffic that is duplicated is from VM to router/external network. (via default route to router) As far as I could see, this issue does not appear with private networks, since all MAC addresses are known to OVN/Neutron.
Versions: Nova/Neutron: Yoga ovs-vswitchd (Open vSwitch) 3.1.4 ovn-northd 22.12.1 kernel 5.14.0-284.30.1.el9_2.x86_64
Best, Daniel
[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
Hi,
what I would like to achieve is for a VM to communicate with IPs not available in neutron/OVN to forward the traffic to the router directly. Is there a way to tell neutron/OVN the MAC address of the router directly?
We also just found the option to set localnet_learn_fdb=true on the provnet port, which was also introduced to neutron in 2023.2 with [1] and enables the southbound to learn the MAC of the router in the FDB table. Another possibility would be to create the entry in the southbound DB manually in the FDB table.
It appears, we hit the issue mentioned in [2]. A fix for us is either enabling port_security and/or adding localnet_learn_fdb=true.
Thank you for the help, Daniel
[1] https://review.opendev.org/c/openstack/neutron/+/877675https://review.opendev.org/c/openstack/neutron/+/877675 [2] https://bugs.launchpad.net/neutron/+bug/2012069 ________________________________ From: Rodolfo Alonso Hernandez ralonsoh@redhat.com Sent: Friday, February 9, 2024 09:23 To: Daniel Failing daniel.failing@cern.ch Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [Neutron][OVN] Flat routed provider network has external traffic broadcasted to all VMs
Hello Daniel:
OVN uses GW ports (on GW chassis) to egress traffic outside the OVN network. If you are using a flat/vlan network, your configuration will work but only because both the VM (that has a port bound to the OVN mech driver and thus stored in the OVN SB database) and the laptop are in the same broadcast domain. OVN is broadcasting the VM traffic because OVN doesn't know what the laptop IP is.
This configuration is not supported in OpenStack: any access to an external IP should be done through a GW port.
Regards.
On Thu, Feb 8, 2024 at 4:58 PM Daniel Failing <daniel.failing@cern.chmailto:daniel.failing@cern.ch> wrote: Hello Rodolfo,
For DC router, yes, I mean datacenter router, in this case not managed by neutron. We are communicating to external resources through said router. It looks like this: {HV1, HV2, HV3} - switch - router (- end user device, e.g. laptop1). In the hypervisor/ovs: {tap,...} - br-int - patchport - br-eth0 - eth0 There is no OVN router involved in the external network.
The hypervisors and the router are in the same subnet, which is configured in neutron as one subnet. The laptop would be outside of neutron scope, while the router IP is configured as the gateway IP of the subnet.
Best, Daniel ________________________________ From: Rodolfo Alonso Hernandez <ralonsoh@redhat.commailto:ralonsoh@redhat.com> Sent: Thursday, February 8, 2024 16:24 To: Daniel Failing <daniel.failing@cern.chmailto:daniel.failing@cern.ch> Cc: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Neutron][OVN] Flat routed provider network has external traffic broadcasted to all VMs
Hello Daniel:
* I'm not very good with acronyms: what is DC router? Datacenter I guess, right? * How are you communicating to external resources? Are you using an OVN router and an external network?
Regards.
On Thu, Feb 8, 2024 at 10:28 AM Daniel Failing <daniel.failing@cern.chmailto:daniel.failing@cern.ch> wrote: Dear all,
I'm currently debugging a (testing) deployment with neutron/OVN, currently most ports in external flat network. So far I'm happy everything can communicate, but we see a strong performance degradation with increasing number of VMs/ports hosted on a hypervisor. The setup is similar to the one described in the routed provider networks docs [1]. For routing between the different segments, we have physical routers, that are not managed by neutron.
Debugging the issue showed traffic originating from the DC router to VMs go perfectly as unicast traffic through OVN. Communicating from the VM to other networks via the DC router, however, will broadcast the traffic to all attached ports on the internal bridge.
Is there a way to tell neutron/OVN to use a specific MAC address for the router and forward the traffic through the patch port instead of broadcasting it? I could not find any documentation of whether I am missing a neutron port or some other configuration on the hypervisor.
Just to make sure: The VM and DC routers have the right ARP information of itself, the traffic that is duplicated is from VM to router/external network. (via default route to router) As far as I could see, this issue does not appear with private networks, since all MAC addresses are known to OVN/Neutron.
Versions: Nova/Neutron: Yoga ovs-vswitchd (Open vSwitch) 3.1.4 ovn-northd 22.12.1 kernel 5.14.0-284.30.1.el9_2.x86_64
Best, Daniel
[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
participants (2)
-
Daniel Failing
-
Rodolfo Alonso Hernandez