[ovn][neutron] RE: OVN BGP Agent query
Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez dalvarez@redhat.com Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar ltomasbo@redhat.com wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq ihtisham.uh@hotmail.com wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment? like openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM IP routes are announced via the router IP(of the that customer) to the leaf switch instead of the IP of the host where the neutron BGP agent runs. And then even if the router fails over, the IP of the router stays the same and thus the BGP route doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it? Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of where 172.24.4.100 is). If there's L2 in the 172.24.4.0 network, the new location of 172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the host IP(GTW) in our switch peers. If that's the case then how do you make sure that during failover of a router, the BGP routes gets updated with the new host IP(where the router failed over to)?
The local FRR running at each node is in charge of exposing the IPs. For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route. In the case of VMs on tenant networks, the traffic is exposed on the node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup, using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz wrote:
Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez dalvarez@redhat.com Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar ltomasbo@redhat.com
wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <
ihtisham.uh@hotmail.com> wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment?
like
openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM
IP routes
are announced via the router IP(of the that customer) to the leaf
switch instead of
the IP of the host where the neutron BGP agent runs. And then even
if the
router fails over, the IP of the router stays the same and thus the
BGP route
doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you
mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it?
Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of
where 172.24.4.100 is).
If there's L2 in the 172.24.4.0 network, the new location of
172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that
everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the
host IP(GTW) in our
switch peers. If that's the case then how do you make sure that
during failover
of a router, the BGP routes gets updated with the new host IP(where
the router
failed over to)?
The local FRR running at each node is in charge of exposing the IPs.
For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route.
In the case of VMs on tenant networks, the traffic is exposed on the
node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup,
using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute
that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
Great to hear!!
Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as: - 10.0.0.5 nexthop 192.168.100.100 - 172.24.4.10 nexthop 192.168.100.100
As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides those route we also advertise? - 10.0.0.5 nexthop 172.24.4.10
Cheers, Luis
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
Hey Luis,
Thanks for your mail.
We have now prepared a first draft. In addition to what Ihtisham already wrote, we need the following options: - to run multiple agents per host (container) because we need more than one BGP session, alternatively there would be the option to do it via frr - we need to be able to filter which networks we announce where, for this we used the address scope from Openstack in the past
To make this possible we have built it in so that it runs through the address scope. To make the address scope usable in ovn-bgp-agent we also patched Neutron so that the address scope is part of router_gateway and router_interface patch-ports. We also announce the networks behind the routers instead of the VM IPs directly.
We added a new driver for this because we don't really need anything from the previous implementation at the moment and we were missing the similarities between the two. Possibly someone has an idea how we could merge this?
It would be nice if you could have a first look at it.
There is still some work to do like adding tests and making Zuul happy but I think it is useful to discuss this with you as early as possible.
You can find the merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861581
Best regards, Luca Czesla
From: Luis Tomas Bolivar ltomasbo@redhat.com Sent: Monday, October 3, 2022 7:45 AM To: Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz Cc: Daniel Alvarez dalvarez@redhat.com; Luca Czesla Luca.Czesla@mail.schwarz; Max André Lamprecht Max.Lamprecht@mail.schwarz; openstack-discuss openstack-discuss@lists.openstack.org Subject: Re: [ovn][neutron] RE: OVN BGP Agent query
On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq <Ihtisham.ul_Haq@mail.schwarzmailto:Ihtisham.ul_Haq@mail.schwarz> wrote: Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez <dalvarez@redhat.commailto:dalvarez@redhat.com> Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar <ltomasbo@redhat.commailto:ltomasbo@redhat.com> wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <ihtisham.uh@hotmail.commailto:ihtisham.uh@hotmail.com> wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment? like openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM IP routes are announced via the router IP(of the that customer) to the leaf switch instead of the IP of the host where the neutron BGP agent runs. And then even if the router fails over, the IP of the router stays the same and thus the BGP route doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it? Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of where 172.24.4.100 is). If there's L2 in the 172.24.4.0 network, the new location of 172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the host IP(GTW) in our switch peers. If that's the case then how do you make sure that during failover of a router, the BGP routes gets updated with the new host IP(where the router failed over to)?
The local FRR running at each node is in charge of exposing the IPs. For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route. In the case of VMs on tenant networks, the traffic is exposed on the node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup, using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
Great to hear!!
Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as: - 10.0.0.5 nexthop 192.168.100.100 - 172.24.4.10 nexthop 192.168.100.100
As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides those route we also advertise? - 10.0.0.5 nexthop 172.24.4.10
Cheers, Luis
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarzhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C0635d1ca0f4d4281fff108daa502667b%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638003726972382016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sFol3dUCKOU6UiSMk4Y2%2BLwXyZRoe83LKbANBq%2Bi7qg%3D&reserved=0>.
-- LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.commailto:ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
Hello Luca!
Awesome patch! See some comments below, though I'll also add some more details on the gerrit side
On Mon, Oct 17, 2022 at 11:12 AM Luca Czesla Luca.Czesla@mail.schwarz wrote:
Hey Luis,
Thanks for your mail.
We have now prepared a first draft. In addition to what Ihtisham already wrote, we need the following options:
- to run multiple agents per host (container) because we need more than
one BGP session, alternatively there would be the option to do it via frr
Nice, this could actually be splitted from the patch into a different one, as it is kind of independent of the new driver, and can be used for other use cases (and merge it almost right away)
- we need to be able to filter which networks we announce where, for this
we used the address scope from Openstack in the past
To make this possible we have built it in so that it runs through the address scope. To make the address scope usable in ovn-bgp-agent we also patched Neutron so that the address scope is part of router_gateway and router_interface patch-ports. We also announce the networks behind the routers instead of the VM IPs directly.
We actually play with something similar to be able to have a simple/initial API here: https://review.opendev.org/c/openstack/neutron/+/797087 (idea was to use port tags instead)
Do you have a link to the modifications to enable this in the neutron side?
We added a new driver for this because we don't really need anything from the previous implementation at the moment and we were missing the similarities between the two. Possibly someone has an idea how we could merge this?
Is the new expose_router_port/withdraw_router_port similar to the previous expose_subnet/withdraw_subnet? Perhaps we just need a different implementation of those?
I was actually playing (after initial discussion with Ihtisha) for options to not only expose VM_IP via HOST_IP, but doing it VM_IP though OVN_GATEWAY_PORT_IP, and then GATEWAY_PORT_IP via HOST_IP. I was playing with "redistribute static" instead of "redistributed kernel" + IP routes (I think I like your approach better).
My idea was to have the existing driver with an option to state if the env is pure L3 (the current support) or L2 (where you work is needed). That is another option to explore, but open to having completely different drivers, as it also makes a lot of sense.
It would be nice if you could have a first look at it.
There is still some work to do like adding tests and making Zuul happy but I think it is useful to discuss this with you as early as possible.
Yeah, no problem, we'll make zull happy after the initial design/idea is discussed. Don't need to waste time on that at this moment
You can find the merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861581
Thanks! I'll leave some more detailed review in there
There is one more general question on that patch. Isn't that exposing the routes in all the nodes where the agent is running instead of where the actual ovn router gateway port is attached? Don't you need to check is the node where you can actually inject the traffic to OVN overlay? Or you are assuming everything is L2 at that point, and it does not matter everyone expose the route as at the end of the day, only the node with the ovn gateway router port will reply to ARPs
Best regards,
Luca Czesla
*From:* Luis Tomas Bolivar ltomasbo@redhat.com *Sent:* Monday, October 3, 2022 7:45 AM *To:* Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz *Cc:* Daniel Alvarez dalvarez@redhat.com; Luca Czesla Luca.Czesla@mail.schwarz; Max André Lamprecht Max.Lamprecht@mail.schwarz; openstack-discuss < openstack-discuss@lists.openstack.org> *Subject:* Re: [ovn][neutron] RE: OVN BGP Agent query
On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq < Ihtisham.ul_Haq@mail.schwarz> wrote:
Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez dalvarez@redhat.com Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar ltomasbo@redhat.com
wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <
ihtisham.uh@hotmail.com> wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment?
like
openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM
IP routes
are announced via the router IP(of the that customer) to the leaf
switch instead of
the IP of the host where the neutron BGP agent runs. And then even
if the
router fails over, the IP of the router stays the same and thus the
BGP route
doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you
mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it?
Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of
where 172.24.4.100 is).
If there's L2 in the 172.24.4.0 network, the new location of
172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that
everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the
host IP(GTW) in our
switch peers. If that's the case then how do you make sure that
during failover
of a router, the BGP routes gets updated with the new host IP(where
the router
failed over to)?
The local FRR running at each node is in charge of exposing the IPs.
For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route.
In the case of VMs on tenant networks, the traffic is exposed on the
node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup,
using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute
that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
Great to hear!!
Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as:
10.0.0.5 nexthop 192.168.100.100
172.24.4.10 nexthop 192.168.100.100
As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides those route we also advertise?
- 10.0.0.5 nexthop 172.24.4.10
Cheers,
Luis
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C0635d1ca0f4d4281fff108daa502667b%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638003726972382016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sFol3dUCKOU6UiSMk4Y2%2BLwXyZRoe83LKbANBq%2Bi7qg%3D&reserved=0
.
--
LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier https://www.datenschutz.schwarz.
Hey Luis,
thanks a lot for the great feedback.
See inline reply please.
Best regards, Luca Czesla
From: Luis Tomas Bolivar ltomasbo@redhat.com Sent: Tuesday, October 18, 2022 9:54 AM To: Luca Czesla Luca.Czesla@mail.schwarz Cc: Daniel Alvarez dalvarez@redhat.com; Max André Lamprecht Max.Lamprecht@mail.schwarz; openstack-discuss openstack-discuss@lists.openstack.org; Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz Subject: Re: [ovn][neutron] RE: OVN BGP Agent query
Hello Luca!
Awesome patch! See some comments below, though I'll also add some more details on the gerrit side
On Mon, Oct 17, 2022 at 11:12 AM Luca Czesla <Luca.Czesla@mail.schwarzmailto:Luca.Czesla@mail.schwarz> wrote: Hey Luis,
Thanks for your mail.
We have now prepared a first draft. In addition to what Ihtisham already wrote, we need the following options: - to run multiple agents per host (container) because we need more than one BGP session, alternatively there would be the option to do it via frr
Nice, this could actually be splitted from the patch into a different one, as it is kind of independent of the new driver, and can be used for other use cases (and merge it almost right away)
Good idea. I opened a separate merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861749
- we need to be able to filter which networks we announce where, for this we used the address scope from Openstack in the past
To make this possible we have built it in so that it runs through the address scope. To make the address scope usable in ovn-bgp-agent we also patched Neutron so that the address scope is part of router_gateway and router_interface patch-ports. We also announce the networks behind the routers instead of the VM IPs directly.
We actually play with something similar to be able to have a simple/initial API here: https://review.opendev.org/c/openstack/neutron/+/797087https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fopenstack%2Fneutron%2F%2B%2F797087&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BwwlZI48MmY4fp5mSCaCDY7xYgN3CbmjaVF5QyfxWtg%3D&reserved=0 (idea was to use port tags instead)
Do you have a link to the modifications to enable this in the neutron side?
I just opened the MR at Neutron. It's pretty much what you do with the port_tags as well. You can find it here: https://review.opendev.org/c/openstack/neutron/+/861719 Is there an advantage of using "port_tags" at this point, or is that simply a catch-all term for metadata? I am fine with both solutions and have no preferences there.
We added a new driver for this because we don't really need anything from the previous implementation at the moment and we were missing the similarities between the two. Possibly someone has an idea how we could merge this?
Is the new expose_router_port/withdraw_router_port similar to the previous expose_subnet/withdraw_subnet? Perhaps we just need a different implementation of those?
I am not sure if it is the right way. I had problems for example with the withdraw_subnet that I could not get the corresponding router port from the lrp port because it was already deleted, so I was missing the reference to the address scope. So if you have a better way to get to the target, I will gladly accept it.
The function is similar only that I didn't want to use the lrp ports for the reason above and we need the update event since initial row.mac is unknown and will only be set to router with the next revision number. I am not sure how we could merge this but I am of course open for ideas. Another idea would be to have another separate watcher that implements only these two events. Maybe that would be the cleanest way?
I was actually playing (after initial discussion with Ihtisha) for options to not only expose VM_IP via HOST_IP, but doing it VM_IP though OVN_GATEWAY_PORT_IP, and then GATEWAY_PORT_IP via HOST_IP. I was playing with "redistribute static" instead of "redistributed kernel" + IP routes (I think I like your approach better).
My idea was to have the existing driver with an option to state if the env is pure L3 (the current support) or L2 (where you work is needed). That is another option to explore, but open to having completely different drivers, as it also makes a lot of sense.
It would be nice if you could have a first look at it.
There is still some work to do like adding tests and making Zuul happy but I think it is useful to discuss this with you as early as possible.
Yeah, no problem, we'll make zull happy after the initial design/idea is discussed. Don't need to waste time on that at this moment
You can find the merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861581https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fx%2Fovn-bgp-agent%2F%2B%2F861581&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M7AF9ux%2FdEg6uxQ9TlJaEkphNV3KjOVLOJO%2F3Qd1P%2F8%3D&reserved=0
Thanks! I'll leave some more detailed review in there
Thank you very much. I will try to work on it tomorrow.
There is one more general question on that patch. Isn't that exposing the routes in all the nodes where the agent is running instead of where the actual ovn router gateway port is attached? Don't you need to check is the node where you can actually inject the traffic to OVN overlay? Or you are assuming everything is L2 at that point, and it does not matter everyone expose the route as at the end of the day, only the node with the ovn gateway router port will reply to ARPs
Yes exactly, we assume that the respective router IPs/Networks are properly filtered via the address scope and that the BGP peers are in the same L2 network as the provider network of the routers. The patch is really only about announcing the tenant networks via the correct router IP. We are not interested in where this IP is located, this task can be done by ARP in the respective L2 network. So everything can be routed locally in the L2 via ARP lookups and every BGP agent that has a foot in the network can announce the networks identically. So in the end we have N times the same route which only differ from whom it was received. Announcing the complete networks makes sense for us as we have less BGP updates and the routers can drop the traffic. The advantage of this is that L3 failover is instant, since no new BGP route needs to be announced. The switching of the router IP to another gateway/chassis is then done by OVN using GARP.
Best regards, Luca Czesla
From: Luis Tomas Bolivar <ltomasbo@redhat.commailto:ltomasbo@redhat.com> Sent: Monday, October 3, 2022 7:45 AM To: Ihtisham ul Haq <Ihtisham.ul_Haq@mail.schwarzmailto:Ihtisham.ul_Haq@mail.schwarz> Cc: Daniel Alvarez <dalvarez@redhat.commailto:dalvarez@redhat.com>; Luca Czesla <Luca.Czesla@mail.schwarzmailto:Luca.Czesla@mail.schwarz>; Max André Lamprecht <Max.Lamprecht@mail.schwarzmailto:Max.Lamprecht@mail.schwarz>; openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [ovn][neutron] RE: OVN BGP Agent query
On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq <Ihtisham.ul_Haq@mail.schwarzmailto:Ihtisham.ul_Haq@mail.schwarz> wrote: Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez <dalvarez@redhat.commailto:dalvarez@redhat.com> Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar <ltomasbo@redhat.commailto:ltomasbo@redhat.com> wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <ihtisham.uh@hotmail.commailto:ihtisham.uh@hotmail.com> wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment? like openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM IP routes are announced via the router IP(of the that customer) to the leaf switch instead of the IP of the host where the neutron BGP agent runs. And then even if the router fails over, the IP of the router stays the same and thus the BGP route doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it? Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of where 172.24.4.100 is). If there's L2 in the 172.24.4.0 network, the new location of 172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the host IP(GTW) in our switch peers. If that's the case then how do you make sure that during failover of a router, the BGP routes gets updated with the new host IP(where the router failed over to)?
The local FRR running at each node is in charge of exposing the IPs. For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route. In the case of VMs on tenant networks, the traffic is exposed on the node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup, using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
Great to hear!!
Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as: - 10.0.0.5 nexthop 192.168.100.100 - 172.24.4.10 nexthop 192.168.100.100
As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides those route we also advertise? - 10.0.0.5 nexthop 172.24.4.10
Cheers, Luis
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarzhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0>.
-- LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.commailto:ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hierhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0.
-- LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.commailto:ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hierhttps://www.datenschutz.schwarz.
On Tue, Oct 18, 2022 at 5:06 PM Luca Czesla Luca.Czesla@mail.schwarz wrote:
Hey Luis,
thanks a lot for the great feedback.
See inline reply please.
Best regards,
Luca Czesla
*From:* Luis Tomas Bolivar ltomasbo@redhat.com *Sent:* Tuesday, October 18, 2022 9:54 AM *To:* Luca Czesla Luca.Czesla@mail.schwarz *Cc:* Daniel Alvarez dalvarez@redhat.com; Max André Lamprecht Max.Lamprecht@mail.schwarz; openstack-discuss < openstack-discuss@lists.openstack.org>; Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz *Subject:* Re: [ovn][neutron] RE: OVN BGP Agent query
Hello Luca!
Awesome patch! See some comments below, though I'll also add some more details on the gerrit side
On Mon, Oct 17, 2022 at 11:12 AM Luca Czesla Luca.Czesla@mail.schwarz wrote:
Hey Luis,
Thanks for your mail.
We have now prepared a first draft. In addition to what Ihtisham already wrote, we need the following options:
- to run multiple agents per host (container) because we need more than
one BGP session, alternatively there would be the option to do it via frr
Nice, this could actually be splitted from the patch into a different one, as it is kind of independent of the new driver, and can be used for other use cases (and merge it almost right away)
Good idea. I opened a separate merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861749
Great! Thanks!
- we need to be able to filter which networks we announce where, for this
we used the address scope from Openstack in the past
To make this possible we have built it in so that it runs through the address scope. To make the address scope usable in ovn-bgp-agent we also patched Neutron so that the address scope is part of router_gateway and router_interface patch-ports. We also announce the networks behind the routers instead of the VM IPs directly.
We actually play with something similar to be able to have a simple/initial API here: https://review.opendev.org/c/openstack/neutron/+/797087 https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fopenstack%2Fneutron%2F%2B%2F797087&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BwwlZI48MmY4fp5mSCaCDY7xYgN3CbmjaVF5QyfxWtg%3D&reserved=0 (idea was to use port tags instead)
Do you have a link to the modifications to enable this in the neutron side?
I just opened the MR at Neutron. It's pretty much what you do with the port_tags as well. You can find it here: https://review.opendev.org/c/openstack/neutron/+/861719
Is there an advantage of using "port_tags" at this point, or is that simply a catch-all term for metadata? I am fine with both solutions and have no preferences there.
No, there is no advantage of using port_tags, in fact after discussing it with the neutron community it was stated that that was not included on purpose as it is a field for layers that work on top of neutron, not below (OVN). So, I think your approach makes more sense.
We added a new driver for this because we don't really need anything from the previous implementation at the moment and we were missing the similarities between the two. Possibly someone has an idea how we could merge this?
Is the new expose_router_port/withdraw_router_port similar to the previous expose_subnet/withdraw_subnet? Perhaps we just need a different implementation of those?
I am not sure if it is the right way. I had problems for example with the withdraw_subnet that I could not get the corresponding router port from the lrp port because it was already deleted, so I was missing the reference to the address scope. So if you have a better way to get to the target, I will gladly accept it.
Right, I think that was the reason why I used an in-memory dict for that (ovn_local_cr_lrps and ovn_local_lrps)
The function is similar only that I didn't want to use the lrp ports for the reason above and we need the update event since initial row.mac is unknown and will only be set to router with the next revision number. I am not sure how we could merge this but I am of course open for ideas.
Another idea would be to have another separate watcher that implements only these two events. Maybe that would be the cleanest way?
Umm, ok, probably I'm missing something but I did not have problems with row.mac for the SubnetRouterAttachedEvent, but maybe this is related to the withdraw issue.
I was actually playing (after initial discussion with Ihtisha) for options to not only expose VM_IP via HOST_IP, but doing it VM_IP though OVN_GATEWAY_PORT_IP, and then GATEWAY_PORT_IP via HOST_IP. I was playing with "redistribute static" instead of "redistributed kernel" + IP routes (I think I like your approach better).
My idea was to have the existing driver with an option to state if the env is pure L3 (the current support) or L2 (where you work is needed). That is another option to explore, but open to having completely different drivers, as it also makes a lot of sense.
It would be nice if you could have a first look at it.
There is still some work to do like adding tests and making Zuul happy but I think it is useful to discuss this with you as early as possible.
Yeah, no problem, we'll make zull happy after the initial design/idea is discussed. Don't need to waste time on that at this moment
You can find the merge request here: https://review.opendev.org/c/x/ovn-bgp-agent/+/861581 https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fx%2Fovn-bgp-agent%2F%2B%2F861581&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M7AF9ux%2FdEg6uxQ9TlJaEkphNV3KjOVLOJO%2F3Qd1P%2F8%3D&reserved=0
Thanks! I'll leave some more detailed review in there
Thank you very much. I will try to work on it tomorrow.
There is one more general question on that patch. Isn't that exposing the routes in all the nodes where the agent is running instead of where the actual ovn router gateway port is attached? Don't you need to check is the node where you can actually inject the traffic to OVN overlay? Or you are assuming everything is L2 at that point, and it does not matter everyone expose the route as at the end of the day, only the node with the ovn gateway router port will reply to ARPs
Yes exactly, we assume that the respective router IPs/Networks are properly filtered via the address scope and that the BGP peers are in the same L2 network as the provider network of the routers. The patch is really only about announcing the tenant networks via the correct router IP. We are not interested in where this IP is located, this task can be done by ARP in the respective L2 network. So everything can be routed locally in the L2 via ARP lookups and every BGP agent that has a foot in the network can announce the networks identically. So in the end we have N times the same route which only differ from whom it was received. Announcing the complete networks makes sense for us as we have less BGP updates and the routers can drop the traffic. The advantage of this is that L3 failover is instant, since no new BGP route needs to be announced. The switching of the router IP to another gateway/chassis is then done by OVN using GARP.
Agree! It won't work for pure L3 domains, but works with L2/ARP as you mention. Thanks for confirming!
Cheers, Luis
Best regards,
Luca Czesla
*From:* Luis Tomas Bolivar ltomasbo@redhat.com *Sent:* Monday, October 3, 2022 7:45 AM *To:* Ihtisham ul Haq Ihtisham.ul_Haq@mail.schwarz *Cc:* Daniel Alvarez dalvarez@redhat.com; Luca Czesla < Luca.Czesla@mail.schwarz>; Max André Lamprecht Max.Lamprecht@mail.schwarz; openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [ovn][neutron] RE: OVN BGP Agent query
On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq < Ihtisham.ul_Haq@mail.schwarz> wrote:
Hi Luis and Daniel,
Please see inline response.
From: Daniel Alvarez Sanchez dalvarez@redhat.com Sent: 29 September 2022 11:37 Subject: Re: OVN BGP Agent query
Hi Ihtisham and Luis,
On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar ltomasbo@redhat.com
wrote:
Some comments and questions inline
On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <
ihtisham.uh@hotmail.com> wrote:
Hi Luis,
Thanks for your work on the OVN BGP Agent. We are planning to use it in our OVN deployment, but have a question regarding it.
Great to hear! Can you share a bit more info about this environment?
like
openstack version, target workload, etc.
We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former.
The way our current setup with ML2/OVS works is that our customer VM
IP routes
are announced via the router IP(of the that customer) to the leaf
switch instead of
the IP of the host where the neutron BGP agent runs. And then even
if the
router fails over, the IP of the router stays the same and thus the
BGP route
doesn't need to be updated.
Is this with Neutron Dynamic Routing? When you say Router IP, do you
mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?
Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address.
If the router fails over, the route needs to be updated, doesn't it?
Same IP, but exposed in the new location of the router?
Correct.
The route to the tenant network doesn't change, ie. 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of
where 172.24.4.100 is).
If there's L2 in the 172.24.4.0 network, the new location of
172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".
Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned.
In the case of Neutron Dynamic Routing, there's no assumption that
everything is L3 so GARPs are needed to learn the new location.
We see that the routes are announced by the ovn-bgp-agent via the
host IP(GTW) in our
switch peers. If that's the case then how do you make sure that
during failover
of a router, the BGP routes gets updated with the new host IP(where
the router
failed over to)?
The local FRR running at each node is in charge of exposing the IPs.
For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route.
In the case of VMs on tenant networks, the traffic is exposed on the
node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location
Can we accomplish the same route advertisement as our ML2/OVS setup,
using the ovn-bgp-agent?
I think this is technically possible, and perhaps you want to contribute
that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?
Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case.
Great to hear!!
Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as:
10.0.0.5 nexthop 192.168.100.100
172.24.4.10 nexthop 192.168.100.100
As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides those route we also advertise?
- 10.0.0.5 nexthop 172.24.4.10
Cheers,
Luis
-- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0
.
--
LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0 .
--
LUIS TOMÁS BOLÍVAR Principal Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier https://www.datenschutz.schwarz.
participants (3)
-
Ihtisham ul Haq
-
Luca Czesla
-
Luis Tomas Bolivar