<div dir="ltr"><div dir="ltr"><div>Hello Luca! <br></div><div><br></div><div>Awesome patch! See some comments below, though I'll also add some more details on the gerrit side<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 17, 2022 at 11:12 AM Luca Czesla <Luca.Czesla@mail.schwarz> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055">
<div style="overflow-wrap: break-word;" lang="DE">
<div class="m_2729431729641458778WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hey Luis,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for your mail.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">We have now prepared a first draft. In addition to what Ihtisham already wrote, we need the following options:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">- to run multiple agents per host (container) because we need more than one BGP session, alternatively there would be the option to do it via frr</span></p></div></div></div></blockquote><div><br></div><div>Nice, this could actually be splitted from the patch into a different one, as it is kind of independent of the new driver, and can be used for other use cases (and merge it almost right away)</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">- we need to be able to filter which networks we announce where, for this we used the address scope from Openstack in the past</span></p></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">To make this possible we have built it in so that it runs through the address scope. To make the address scope usable in ovn-bgp-agent we also patched Neutron so that the address scope
is part of router_gateway and router_interface patch-ports. We also announce the networks behind the routers instead of the VM IPs directly.</span></p></div></div></div></blockquote><div><br></div><div><div>We actually play with something similar to be able to have a
simple/initial API here:
<a href="https://review.opendev.org/c/openstack/neutron/+/797087">https://review.opendev.org/c/openstack/neutron/+/797087</a> (idea was to use
port tags instead)</div><div><br></div><div>Do you have a link to the modifications to enable this in the neutron side?</div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">We added a new driver for this because we don't really need anything from the previous implementation at the moment and we were missing the similarities between the two. Possibly someone
has an idea how we could merge this?</span></p></div></div></div></blockquote><div><br></div><div>Is the new expose_router_port/withdraw_router_port similar to the previous expose_subnet/withdraw_subnet? Perhaps we just need a different implementation of those? <br></div><div><br></div><div>I was actually playing (after initial discussion with <span><span aria-label="Ihtisham ul haq (From your Google Contacts)">Ihtisha) for options to not only expose VM_IP via HOST_IP, but doing it VM_IP though OVN_GATEWAY_PORT_IP, and then GATEWAY_PORT_IP via HOST_IP. I was playing with "redistribute static" instead of "redistributed kernel" + IP routes (I think I like your approach better).</span></span></div><div><span><span aria-label="Ihtisham ul haq (From your Google Contacts)"><br></span></span></div><div><span><span aria-label="Ihtisham ul haq (From your Google Contacts)">My idea was to have the existing driver with an option to state if the env is pure L3 (the current support) or L2 (where you work is needed). That is another option to explore, but open to having completely different drivers, as it also makes a lot of sense.<br> </span></span></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">It would be nice if you could have a first look at it.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">There is still some work to do like adding tests and making Zuul happy but I think it is useful to discuss this with you as early as possible.</span></p></div></div></div></blockquote><div><br></div><div>Yeah, no problem, we'll make zull happy after the initial design/idea is discussed. Don't need to waste time on that at this moment <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">You can find the merge request here: <a href="https://review.opendev.org/c/x/ovn-bgp-agent/+/861581" target="_blank">https://review.opendev.org/c/x/ovn-bgp-agent/+/861581</a></span></p></div></div></div></blockquote><div><br></div><div>Thanks! I'll leave some more detailed review in there</div><div><br></div><div>There is one more general question on that patch. Isn't that exposing the routes in all the nodes where the agent is running instead of where the actual ovn router gateway port is attached? Don't you need to check is the node where you can actually inject the traffic to OVN overlay? Or you are assuming everything is L2 at that point, and it does not matter everyone expose the route as at the end of the day, only the node with the ovn gateway router port will reply to ARPs</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3178419963446936055"><div style="overflow-wrap: break-word;" lang="DE"><div class="m_2729431729641458778WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span>Luca Czesla<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Luis Tomas Bolivar <<a href="mailto:ltomasbo@redhat.com" target="_blank">ltomasbo@redhat.com</a>> <br>
<b>Sent:</b> Monday, October 3, 2022 7:45 AM<br>
<b>To:</b> Ihtisham ul Haq <Ihtisham.ul_Haq@mail.schwarz><br>
<b>Cc:</b> Daniel Alvarez <<a href="mailto:dalvarez@redhat.com" target="_blank">dalvarez@redhat.com</a>>; Luca Czesla <Luca.Czesla@mail.schwarz>; Max André Lamprecht <Max.Lamprecht@mail.schwarz>; openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [ovn][neutron] RE: OVN BGP Agent query<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq <<a href="mailto:Ihtisham.ul_Haq@mail.schwarz" target="_blank">Ihtisham.ul_Haq@mail.schwarz</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi Luis and Daniel,<br>
<br>
Please see inline response.<br>
<br>
> From: Daniel Alvarez Sanchez <<a href="mailto:dalvarez@redhat.com" target="_blank">dalvarez@redhat.com</a>><br>
> Sent: 29 September 2022 11:37<br>
> Subject: Re: OVN BGP Agent query<br>
><br>
> Hi Ihtisham and Luis,<br>
><br>
> On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar <<a href="mailto:ltomasbo@redhat.com" target="_blank">ltomasbo@redhat.com</a>> wrote:<br>
> > Some comments and questions inline<br>
> ><br>
> > On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <<a href="mailto:ihtisham.uh@hotmail.com" target="_blank">ihtisham.uh@hotmail.com</a>> wrote:<br>
> > > Hi Luis,<br>
> > ><br>
> > > Thanks for your work on the OVN BGP Agent. We are planning<br>
> > > to use it in our OVN deployment, but have a question regarding it.<br>
> ><br>
> > Great to hear! Can you share a bit more info about this environment? like<br>
> > openstack version, target workload, etc.<br>
<br>
We plan to use this with Yoga version. Our workload consist of enterprise users<br>
with VMs running on Openstack and connected to their enterprise network via<br>
transfer network(to which the customer neutron router is attached to).<br>
And we also have public workload but with the ovn-bgp we only want to<br>
we want to advertise the former.<br>
<br>
> ><br>
> > > The way our current setup with ML2/OVS works is that our customer VM IP routes<br>
> > > are announced via the router IP(of the that customer) to the leaf switch instead of<br>
> > > the IP of the host where the neutron BGP agent runs. And then even if the<br>
> > > router fails over, the IP of the router stays the same and thus the BGP route<br>
> > > doesn't need to be updated.<br>
> ><br>
> > Is this with Neutron Dynamic Routing? When you say Router IP, do you mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)?<br>
<br>
Yes, that's with Neutron DR agent, and I meant virtual neutron router with<br>
IP from the provider network. We announce IPs of our tenant network via the<br>
virtual routers external address.<br>
<br>
> > If the router fails over, the route needs to be updated, doesn't it? Same IP, but exposed in the new location of the router?<br>
<br>
Correct.<br>
<br>
> The route to the tenant network doesn't change, ie.<br>
> 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of where 172.24.4.100 is).<br>
> If there's L2 in the 172.24.4.0 network, the new location of 172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation".<br>
<br>
Right, in our case we have a stretched L2 transfer network(mentioned above)<br>
to which our gateway nodes and customer routers are connected to, so we can<br>
advertise the IPs from the tenant network via the virtual router external IP<br>
and thus the location of the router isn't relevant in case of failover as its<br>
address will be relearned.<br>
<br>
> In the case of Neutron Dynamic Routing, there's no assumption that everything is L3 so GARPs are needed to learn the new location.<br>
><br>
> > > We see that the routes are announced by the ovn-bgp-agent via the host IP(GTW) in our<br>
> > > switch peers. If that's the case then how do you make sure that during failover<br>
> > > of a router, the BGP routes gets updated with the new host IP(where the router<br>
> > > failed over to)?<br>
> ><br>
> > The local FRR running at each node is in charge of exposing the IPs. For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route.<br>
> > In the case of VMs on tenant networks, the traffic is exposed on the node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw
the exposed routes on the old node, and re-advertise them on the new router IP location<br>
> ><br>
> > > Can we accomplish the same route advertisement as our ML2/OVS setup, using the ovn-bgp-agent?<br>
><br>
> I think this is technically possible, and perhaps you want to contribute that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing?<br>
<br>
Sounds good, our plan currently is to add this to the ovn-bgp-agent,<br>
so we can announce our tenant routes via virtual routers external address on<br>
a stretched L2 network, to make it work with our use case.<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Great to hear!!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just to make it clear, the ovn-bgp-agent current solution is to expose the tenant VM IPs through the host that has the OVN router gateway port, so for example, if the VM IP (10.0.0.5) is connected to the neutron virtual router, which in
turns is connected to your provider network (your transfer network) with IP 172.24.4.10, and hosted in a physical server with IP 192.168.100.100, the route will be exposed as:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- 10.0.0.5 nexthop 192.168.100.100<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- 172.24.4.10 nexthop 192.168.100.100<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">As we are using FRR config "redistributed connected". As the traffic to the tenant networks needs to be injected into the OVN overlay through the gateway node hosting that ovn virtual router gateway port (cr-lrp), would it be ok if, besides
those route we also advertise?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- 10.0.0.5 nexthop 172.24.4.10<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Luis<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><br>
<br>
--<br>
Ihtisham ul Haq<br>
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C0635d1ca0f4d4281fff108daa502667b%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638003726972382016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sFol3dUCKOU6UiSMk4Y2%2BLwXyZRoe83LKbANBq%2Bi7qg%3D&reserved=0" target="_blank">https://www.datenschutz.schwarz</a>>.<u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">LUIS TOMÁS BOLÍVAR<br>
Principal Software Engineer<br>
Red Hat<br>
Madrid, Spain<br>
<a href="mailto:ltomasbo@redhat.com" target="_blank">ltomasbo@redhat.com</a> <br>
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<p style="font-family:Calibri;font-size:11pt">Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte
unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie
<a href="https://www.datenschutz.schwarz" target="_blank">hier</a>.</p>
</div>
</div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>LUIS TOMÁS BOLÍVAR<br>Principal Software Engineer<br>Red Hat<br>Madrid, Spain<br><a href="mailto:ltomasbo@redhat.com" target="_blank">ltomasbo@redhat.com</a> <br> <br></div></div></div></div>