Hi Dan,
Red Hat has begun gathering a team of engineers to add OpenStack support for BGP dynamic routing using the Free Range Routing (FRR) set of daemons. Acting as a technical lead for the project, I led one session in the TripleO room to discuss the installer components and two sessions in the Neutron room to discuss BGP routing with OVN, and BGP EVPN with OVN.
There may be quite a lot of overlap with what we (Ericsson) are working on right now, we would be really interested in your long term vision and also the details of your plans.
There will likely be opportunities to leverage APIs and contribute to existing work being done with Neutron Dynamic Routing, BGPVPN, and other work being done to implement BGP EVPN. We would like to collaborate with Ericsson and others and come up with a solution that fits us all!
There are a few related specs proposed already. Below are two links that may be the most relevant to you. All your input is welcome. BFD: https://review.opendev.org/c/openstack/neutron-specs/+/767337 BGP: https://review.opendev.org/c/openstack/neutron-specs/+/783791
BGP may be used for equal-cost multipath (ECMP) load balancing of outbound links, and bi-directional forwarding detection (BFD) for resiliency to ensure that a path provides connectivity.
BGP may also be used for routing inbound traffic to provider network IPs or floating IPs for instance connectivity.
I believe we also share these two use cases - with some caveats, please see below.
The Compute nodes will run FRR to advertise routes to the local VM IPs or floating IPs hosted on the node. FRR has a daemon named Zebra that is responsible for exchanging routes between routing daemons such as BGP and the kernel. The redistribute connected statement in the FRR configuration will cause local IP addresses on the host to be advertised via BGP. Floating IP addresses are attached to a loopback interface in a namespace, so they will be redistributed using this method. Changes to OVN will be required to ensure provider network IPs assigned to VMs will be assigned to a loopback interface in a namespace in a similar fashion.
Am I getting it right that your primary objective is to route the traffic directly to the hypervisors and there hoist it to the tunnel networks? Some of the links in your email also gave me the impression that occasionaly you'd want to route the traffic to a neutron router's gateway port. Is that right? In which cases? Currently neutron-dynamic-routing advertises routes with their nexthop being the router's gw port. We have a use case for arbitrary VM ports being the nexthop. And you seem to have a use case for the hypervisor being the nexthop. Maybe we could come up with an extension of the n-d-r API that can express these variations... Similar thoughts could be applied to the BFD proposal too.
In the further development of the proof-of-concept, how much do you plan to make this API driven? The PoC seems to be reacting to port binding events, but most other information (peers, filters, maybe nexthops) seem to be coming from TripleO deployed configuration and not from the API. How would you like this to look in the long term?
BGP EVPN with multitenancy will require separate VRFs per tenant. This will allow separate routing tables to be maintained, and allow for overlapping IP addresses for different Neutron tenant networks. FRR may have the capability to utilize a single BGP peering session to combine advertisements for all these VRFs, but there is still work to be done to prototype this design. This may result in more efficient BGP dynamic updates, and could potentially make troubleshooting more straightforward.
BGP to API endpoints and BGPVPN related things are not on our plate right now. However support in Neutron for VRFs could be interesting to us too. Thanks for the great writeup! Cheers, Bence Romsics irc: rubasov Ericsson Software Technology