[neutron] OVN and dynamic routing

Tony Liu tonyliu0592 at hotmail.com
Tue Oct 12 16:04:55 UTC 2021

I wonder, since there is already VXLAN based L2 across multiple racks
(which means you are not looking for pure L3 solution),
while keep tenant network multi-subnet on L3 for EW traffic,
why not have external network also on L2 and stretched on multiple racks
for NS traffic, assuming you are using distributed FIP?

From: Francois <rigault.francois at gmail.com>
Sent: October 12, 2021 07:03 AM
To: openstack-discuss
Subject: [neutron] OVN and dynamic routing

Hello Neutron!
I am looking into running stacks with OVN on a leaf-spine network, and
have some floating IPs routed between racks.

Basically each rack is assigned its own set of subnets.
Some VLANs are stretched across all racks: the provisioning VLAN used
by tripleo to deploy the stack, and the VLANs for the controllers API
IPs. However, each tenant subnet is local to a rack: for example each
OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its
own rack. Traffic between 2 racks is sent to a spine, and leaves and
spines run some eVPN-like thing: each pair of ToR is a vtep, traffic
is encapsulated as VXLAN, and routes between vteps are exchanged with

I am looking into supporting floating IPs in there: I expect floating
IPs to be able to move between racks, as such I am looking into
publishing the route for a FIP towards an hypervisor, through BGP.
Each fip is a /32 subnet with an hypervisor tenant's IP as next hop.

It seems there are several ideas to achieve this (it was discussed
[before][1] in ovs conference)
- using [neutron-dynamic-routing][2] - that seems to have some gaps
for OVN. It uses os-ken to talk to switches and exchange routes
- using [OVN BGP agent][3] that uses FRR, it seems there is a related
[RFE][4] for integration in tripleo

There is btw also a [BGPVPN][5] project (it does not match my usecase
as far as I tried to understand it) that also has some code that talks
BGP to switches, already integrated in tripleo.

For my tests, I was able to use the neutron-dynamic-routing project
(almost) as documented, with a few changes:
- for traffic going from VMs to outside the stack, the hypervisor was
trying to resolve the "gateway of fIPs" with ARP request which does
not make any sense. I created a dummy port with the mac address of the
virtual router of the switches:
$ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml
- Fixed IP Addresses:
- ip_address:
subnet_id: 8f37
ID: 4028
MAC Address: 00:1c:73:00:00:11
Name: lagw
Status: DOWN
this prevent the hypervisor to send ARP requests to a non existent gateway
- for traffic coming back, we start the neutron-bgp-dragent agent on
the controllers. We create the right bgp speaker, peers, etc.
- neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it
selects fips and join with ports owned by a "floatingip_agent_gateway"
which does not exist on OVN. We can define ourselves some ports so
that the dragent is able to find the tenant IP of a host:
openstack port create --network provider --device-owner
network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip
ip-address= ag2
- when creating a floating IP and assigning a port to it, Neutron
reads changes from OVN SB and fills the binding information into the
$ openstack port show -c binding_host_id `openstack floating ip show -f value -c port_id`
| Field | Value |
| binding_host_id | cpu35d.cloud |
this allows the dragent to publish the route for the fip
$ openstack bgp speaker list advertised routes bgpspeaker
| Destination | Nexthop |
| | |
- traffic reaches the hypervisor but (for reason I don't understand) I
had to add a rule
$ ip rule
0: from all lookup local
32765: from all iif vlan1234 lookup ovn
32766: from all lookup main
32767: from all lookup default
$ ip route show table ovn dev vlan1234 scope link
so that the traffic coming for the fip is not immediately discarded by
the hypervisor (it's not an ideal solution but it is a workaround that
makes my one fIP work!)

So all in all it seems it would be possible to use the
neutron-dynamic-routing agent, with some minor modifications (eg: to
also publish the fip of the OVN L3 gateway router).

I am wondering whether I have overlooked anything, and if such kind of
deployment (OVN + neutron dynamic routing or similar) is already in
use somewhere. Does it make sense to have a RFE for better integration
between OVN and neutron-dynamic-routing?


[1]: https://www.openvswitch.org/support/ovscon2020/slides/OVS-CONF-2020-OVN-WITH-DYNAMIC-ROUTING.pdf
[2]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html
[3]: https://ltomasbo.wordpress.com/2021/02/04/openstack-networking-with-bgp/
[4]: https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/triplo-bgp-frrouter.html
[5]: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/neutron-bgpvpn.yaml

More information about the openstack-discuss mailing list