Open Stack

On Tue, Oct 18, 2022 at 5:06 PM Luca Czesla <Luca.Czesla at mail.schwarz>
wrote:

> Hey Luis,
>
>
>
> thanks a lot for the great feedback.
>
>
>
> See inline reply please.
>
>
>
> Best regards,
>
> Luca Czesla
>
>
>
>
>
> *From:* Luis Tomas Bolivar <ltomasbo at redhat.com>
> *Sent:* Tuesday, October 18, 2022 9:54 AM
> *To:* Luca Czesla <Luca.Czesla at mail.schwarz>
> *Cc:* Daniel Alvarez <dalvarez at redhat.com>; Max André Lamprecht
> <Max.Lamprecht at mail.schwarz>; openstack-discuss <
> openstack-discuss at lists.openstack.org>; Ihtisham ul Haq
> <Ihtisham.ul_Haq at mail.schwarz>
> *Subject:* Re: [ovn][neutron] RE: OVN BGP Agent query
>
>
>
> Hello Luca!
>
>
>
> Awesome patch! See some comments below, though I'll also add some more
> details on the gerrit side
>
>
>
> On Mon, Oct 17, 2022 at 11:12 AM Luca Czesla <Luca.Czesla at mail.schwarz>
> wrote:
>
> Hey Luis,
>
>
>
> Thanks for your mail.
>
>
>
> We have now prepared a first draft. In addition to what Ihtisham already
> wrote, we need the following options:
>
> - to run multiple agents per host (container) because we need more than
> one BGP session, alternatively there would be the option to do it via frr
>
>
>
> Nice, this could actually be splitted from the patch into a different one,
> as it is kind of independent of the new driver, and can be used for other
> use cases (and merge it almost right away)
>
>
>
>
>
> Good idea. I opened a separate merge request here:
> https://review.opendev.org/c/x/ovn-bgp-agent/+/861749
>

Great! Thanks!

>
>
> - we need to be able to filter which networks we announce where, for this
> we used the address scope from Openstack in the past
>
>
>
> To make this possible we have built it in so that it runs through the
> address scope. To make the address scope usable in ovn-bgp-agent we also
> patched Neutron so that the address scope is part of router_gateway and
> router_interface patch-ports. We also announce the networks behind the
> routers instead of the VM IPs directly.
>
>
>
> We actually play with something similar to be able to have a
> simple/initial API here:
> https://review.opendev.org/c/openstack/neutron/+/797087
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fopenstack%2Fneutron%2F%2B%2F797087&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BwwlZI48MmY4fp5mSCaCDY7xYgN3CbmjaVF5QyfxWtg%3D&reserved=0>
> (idea was to use port tags instead)
>
>
>
> Do you have a link to the modifications to enable this in the neutron side?
>
>
>
>
>
> I just opened the MR at Neutron. It's pretty much what you do with the
> port_tags as well. You can find it here:
> https://review.opendev.org/c/openstack/neutron/+/861719
>
> Is there an advantage of using "port_tags" at this point, or is that
> simply a catch-all term for metadata? I am fine with both solutions and
> have no preferences there.
>
>
>

No, there is no advantage of using port_tags, in fact after discussing it
with the neutron community it was stated that that was not included on
purpose as it is a field for layers that work on top of neutron, not below
(OVN). So, I think your approach makes more sense.


>
> We added a new driver for this because we don't really need anything from
> the previous implementation at the moment and we were missing the
> similarities between the two. Possibly someone has an idea how we could
> merge this?
>
>
>
> Is the new expose_router_port/withdraw_router_port similar to the previous
> expose_subnet/withdraw_subnet? Perhaps we just need a different
> implementation of those?
>
>
>
>
>
> I am not sure if it is the right way. I had problems for example with the
> withdraw_subnet that I could not get the corresponding router port from the
> lrp port because it was already deleted, so I was missing the reference to
> the address scope. So if you have a better way to get to the target, I will
> gladly accept it.
>

Right, I think that was the reason why I used an in-memory dict for that
(ovn_local_cr_lrps and ovn_local_lrps)

>
>
> The function is similar only that I didn't want to use the lrp ports for
> the reason above and we need the update event since initial row.mac is
> unknown and will only be set to router with the next revision number. I am
> not sure how we could merge this but I am of course open for ideas.
>
> Another idea would be to have another separate watcher that implements
> only these two events. Maybe that would be the cleanest way?
>

Umm,  ok, probably I'm missing something but I did not have problems with
row.mac for the SubnetRouterAttachedEvent, but maybe this is related to the
withdraw issue.

>
>
>
>
> I was actually playing (after initial discussion with Ihtisha) for options
> to not only expose VM_IP via HOST_IP, but doing it VM_IP though
> OVN_GATEWAY_PORT_IP, and then GATEWAY_PORT_IP via HOST_IP. I was playing
> with "redistribute static" instead of "redistributed kernel" + IP routes (I
> think I like your approach better).
>
>
>
> My idea was to have the existing driver with an option to state if the env
> is pure L3 (the current support) or L2 (where you work is needed). That is
> another option to explore, but open to having completely different drivers,
> as it also makes a lot of sense.
>
>
>
> It would be nice if you could have a first look at it.
>
>
>
> There is still some work to do like adding tests and making Zuul happy but
> I think it is useful to discuss this with you as early as possible.
>
>
>
> Yeah, no problem, we'll make zull happy after the initial design/idea is
> discussed. Don't need to waste time on that at this moment
>
>
>
> You can find the merge request here:
> https://review.opendev.org/c/x/ovn-bgp-agent/+/861581
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.opendev.org%2Fc%2Fx%2Fovn-bgp-agent%2F%2B%2F861581&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M7AF9ux%2FdEg6uxQ9TlJaEkphNV3KjOVLOJO%2F3Qd1P%2F8%3D&reserved=0>
>
>
>
> Thanks! I'll leave some more detailed review in there
>
>
>
>
>
> Thank you very much. I will try to work on it tomorrow.
>
>
>
>
>
> There is one more general question on that patch. Isn't that exposing the
> routes in all the nodes where the agent is running instead of where the
> actual ovn router gateway port is attached? Don't you need to check is the
> node where you can actually inject the traffic to OVN overlay? Or you are
> assuming everything is L2 at that point, and it does not matter everyone
> expose the route as at the end of the day, only the node with the ovn
> gateway router port will reply to ARPs
>
>
>
>
>
> Yes exactly, we assume that the respective router IPs/Networks are
> properly filtered via the address scope and that the BGP peers are in the
> same L2 network as the provider network of the routers. The patch is really
> only about announcing the tenant networks via the correct router IP. We are
> not interested in where this IP is located, this task can be done by ARP in
> the respective L2 network. So everything can be routed locally in the L2
> via ARP lookups and every BGP agent that has a foot in the network can
> announce the networks identically. So in the end we have N times the same
> route which only differ from whom it was received. Announcing the complete
> networks makes sense for us as we have less BGP updates and the routers can
> drop the traffic. The advantage of this is that L3 failover is instant,
> since no new BGP route needs to be announced. The switching of the router
> IP to another gateway/chassis is then done by OVN using GARP.
>

Agree! It won't work for pure L3 domains, but works with L2/ARP as you
mention. Thanks for confirming!

Cheers,
Luis


>
>
>
>
>
> Best regards,
>
> Luca Czesla
>
>
>
> *From:* Luis Tomas Bolivar <ltomasbo at redhat.com>
> *Sent:* Monday, October 3, 2022 7:45 AM
> *To:* Ihtisham ul Haq <Ihtisham.ul_Haq at mail.schwarz>
> *Cc:* Daniel Alvarez <dalvarez at redhat.com>; Luca Czesla <
> Luca.Czesla at mail.schwarz>; Max André Lamprecht <Max.Lamprecht at mail.schwarz>;
> openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [ovn][neutron] RE: OVN BGP Agent query
>
>
>
>
>
>
>
> On Fri, Sep 30, 2022 at 6:20 PM Ihtisham ul Haq <
> Ihtisham.ul_Haq at mail.schwarz> wrote:
>
> Hi Luis and Daniel,
>
> Please see inline response.
>
> > From: Daniel Alvarez Sanchez <dalvarez at redhat.com>
> > Sent: 29 September 2022 11:37
> > Subject: Re: OVN BGP Agent query
> >
> > Hi Ihtisham and Luis,
> >
> > On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar <ltomasbo at redhat.com>
> wrote:
> > > Some comments and questions inline
> > >
> > > On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <
> ihtisham.uh at hotmail.com> wrote:
> > > > Hi Luis,
> > > >
> > > > Thanks for your work on the OVN BGP Agent. We are planning
> > > > to use it in our OVN deployment, but have a question regarding it.
> > >
> > > Great to hear! Can you share a bit more info about this environment?
> like
> > > openstack version, target workload, etc.
>
> We plan to use this with Yoga version. Our workload consist of enterprise
> users
> with VMs running on Openstack and connected to their enterprise network via
> transfer network(to which the customer neutron router is attached to).
> And we also have public workload but with the ovn-bgp we only want to
> we want to advertise the former.
>
> > >
> > > > The way our current setup with ML2/OVS works is that our customer VM
> IP routes
> > > > are announced via the router IP(of the that customer) to the leaf
> switch instead of
> > > > the IP of the host where the neutron BGP agent runs. And then even
> if the
> > > > router fails over, the IP of the router stays the same and thus the
> BGP route
> > > > doesn't need to be updated.
> > >
> > > Is this with Neutron Dynamic Routing? When you say Router IP, do you
> mean the virtual neutron router and its IP associated with the provider
> network? What type of IPs are you announcing with BGP? IPs on provider
> network or on tenant networks (or both)?
>
> Yes, that's with Neutron DR agent, and I meant virtual neutron router with
> IP from the provider network. We announce IPs of our tenant network via the
> virtual routers external address.
>
> > > If the router fails over, the route needs to be updated, doesn't it?
> Same IP, but exposed in the new location of the router?
>
> Correct.
>
> > The route to the tenant network doesn't change, ie.
> > 192.168.0.0 via 172.24.4.100  (this route remains the same regardless of
> where 172.24.4.100 is).
> > If there's L2 in the 172.24.4.0 network, the new location of
> 172.24.4.100 will be learnt via GARP announcement. In our case, this won't
> happen as we don't have L2 so we expose directly connected routes to
> overcome this "limitation".
>
> Right, in our case we have a stretched L2 transfer network(mentioned above)
> to which our gateway nodes and customer routers are connected to, so we can
> advertise the IPs from the tenant network via the virtual router external
> IP
> and thus the location of the router isn't relevant in case of failover as
> its
> address will be relearned.
>
> > In the case of Neutron Dynamic Routing, there's no assumption that
> everything is L3 so GARPs are needed to learn the new location.
> >
> > > > We see that the routes are announced by the ovn-bgp-agent via the
> host IP(GTW) in our
> > > > switch peers. If that's the case then how do you make sure that
> during failover
> > > > of a router, the BGP routes gets updated with the new host IP(where
> the router
> > > > failed over to)?
> > >
> > > The local FRR running at each node is in charge of exposing the IPs.
> For the IPs on the provider network, the traffic is directly exposed where
> the VMs are, without having to go through the virtual router, so a router
> failover won't change the route.
> > > In the case of VMs on tenant networks, the traffic is exposed on the
> node where the virtual router gateway port is associated (I suppose this is
> what you refer to with router IP). In the case of a failover the agent is
> in charge of making FRR to withdraw the exposed routes on the old node, and
> re-advertise them on the new router IP location
> > >
> > > > Can we accomplish the same route advertisement as our ML2/OVS setup,
> using the ovn-bgp-agent?
> >
> > I think this is technically possible, and perhaps you want to contribute
> that functionality or even help integrating the agent as a driver of
> Neutron Dynamic Routing?
>
> Sounds good, our plan currently is to add this to the ovn-bgp-agent,
> so we can announce our tenant routes via virtual routers external address
> on
> a stretched L2 network, to make it work with our use case.
>
>
>
> Great to hear!!
>
>
>
> Just to make it clear, the ovn-bgp-agent current solution is to expose the
> tenant VM IPs through the host that has the OVN router gateway port, so for
> example, if the VM IP (10.0.0.5) is connected to the neutron virtual
> router, which in turns is connected to your provider network (your transfer
> network) with IP 172.24.4.10, and hosted in a physical server with IP
> 192.168.100.100, the route will be exposed as:
>
> - 10.0.0.5 nexthop 192.168.100.100
>
> - 172.24.4.10 nexthop 192.168.100.100
>
>
>
> As we are using FRR config "redistributed connected". As the traffic to
> the tenant networks needs to be injected into the OVN overlay through the
> gateway node hosting that ovn virtual router gateway port (cr-lrp), would
> it be ok if, besides those route we also advertise?
>
> - 10.0.0.5 nexthop 172.24.4.10
>
>
>
> Cheers,
>
> Luis
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Ihtisham ul Haq
> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für
> die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht
> der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich
> in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie
> hier<https://www.datenschutz.schwarz
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0>
> >.
>
>
>
> --
>
> LUIS TOMÁS BOLÍVAR
> Principal Software Engineer
> Red Hat
> Madrid, Spain
> ltomasbo at redhat.com
>
>
> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für
> die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht
> der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich
> in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie
> hier
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datenschutz.schwarz%2F&data=05%7C01%7C%7C1ad6d332cbd94ebf95e408dab0ddf376%7Cd04f47175a6e4b98b3f96918e0385f4c%7C0%7C0%7C638016764570286277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2CQpWJ1F4e6lQ2gchNCAQxW0EFzapKk6vP3VS6Buch4%3D&reserved=0>
> .
>
>
>
> --
>
> LUIS TOMÁS BOLÍVAR
> Principal Software Engineer
> Red Hat
> Madrid, Spain
> ltomasbo at redhat.com
>
>
>
>
> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für
> die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht
> der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich
> in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie
> hier <https://www.datenschutz.schwarz>.
>


-- 
LUIS TOMÁS BOLÍVAR
Principal Software Engineer
Red Hat
Madrid, Spain
ltomasbo at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221018/83755439/attachment-0001.htm>