Open Stack

I wonder, since there is already VXLAN based L2 across multiple racks
(which means you are not looking for pure L3 solution),
while keep tenant network multi-subnet on L3 for EW traffic,
why not have external network also on L2 and stretched on multiple racks
for NS traffic, assuming you are using distributed FIP?

Thanks!
Tony
________________________________________
From: Francois <rigault.francois at gmail.com>
Sent: October 12, 2021 07:03 AM
To: openstack-discuss
Subject: [neutron] OVN and dynamic routing

Hello Neutron!
I am looking into running stacks with OVN on a leaf-spine network, and
have some floating IPs routed between racks.

Basically each rack is assigned its own set of subnets.
Some VLANs are stretched across all racks: the provisioning VLAN used
by tripleo to deploy the stack, and the VLANs for the controllers API
IPs. However, each tenant subnet is local to a rack: for example each
OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its
own rack. Traffic between 2 racks is sent to a spine, and leaves and
spines run some eVPN-like thing: each pair of ToR is a vtep, traffic
is encapsulated as VXLAN, and routes between vteps are exchanged with
BGP.

I am looking into supporting floating IPs in there: I expect floating
IPs to be able to move between racks, as such I am looking into
publishing the route for a FIP towards an hypervisor, through BGP.
Each fip is a /32 subnet with an hypervisor tenant's IP as next hop.

It seems there are several ideas to achieve this (it was discussed
[before][1] in ovs conference)
- using [neutron-dynamic-routing][2] - that seems to have some gaps
for OVN. It uses os-ken to talk to switches and exchange routes
- using [OVN BGP agent][3] that uses FRR, it seems there is a related
[RFE][4] for integration in tripleo

There is btw also a [BGPVPN][5] project (it does not match my usecase
as far as I tried to understand it) that also has some code that talks
BGP to switches, already integrated in tripleo.

For my tests, I was able to use the neutron-dynamic-routing project
(almost) as documented, with a few changes:
- for traffic going from VMs to outside the stack, the hypervisor was
trying to resolve the "gateway of fIPs" with ARP request which does
not make any sense. I created a dummy port with the mac address of the
virtual router of the switches:
```
$ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml
- Fixed IP Addresses:
- ip_address: 10.64.254.1
subnet_id: 8f37
ID: 4028
MAC Address: 00:1c:73:00:00:11
Name: lagw
Status: DOWN
```
this prevent the hypervisor to send ARP requests to a non existent gateway
- for traffic coming back, we start the neutron-bgp-dragent agent on
the controllers. We create the right bgp speaker, peers, etc.
- neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it
selects fips and join with ports owned by a "floatingip_agent_gateway"
which does not exist on OVN. We can define ourselves some ports so
that the dragent is able to find the tenant IP of a host:
```
openstack port create --network provider --device-owner
network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip
ip-address=10.64.245.102 ag2
```
- when creating a floating IP and assigning a port to it, Neutron
reads changes from OVN SB and fills the binding information into the
port:
```
$ openstack port show -c binding_host_id `openstack floating ip show
10.64.254.177 -f value -c port_id`
+-----------------+----------------------------------------+
| Field | Value |
+-----------------+----------------------------------------+
| binding_host_id | cpu35d.cloud |
+-----------------+----------------------------------------+
```
this allows the dragent to publish the route for the fip
```
$ openstack bgp speaker list advertised routes bgpspeaker
+------------------+---------------+
| Destination | Nexthop |
+------------------+---------------+
| 10.64.254.177/32 | 10.64.245.102 |
+------------------+---------------+
```
- traffic reaches the hypervisor but (for reason I don't understand) I
had to add a rule
```
$ ip rule
0: from all lookup local
32765: from all iif vlan1234 lookup ovn
32766: from all lookup main
32767: from all lookup default
$ ip route show table ovn
10.64.254.177 dev vlan1234 scope link
```
so that the traffic coming for the fip is not immediately discarded by
the hypervisor (it's not an ideal solution but it is a workaround that
makes my one fIP work!)

So all in all it seems it would be possible to use the
neutron-dynamic-routing agent, with some minor modifications (eg: to
also publish the fip of the OVN L3 gateway router).

I am wondering whether I have overlooked anything, and if such kind of
deployment (OVN + neutron dynamic routing or similar) is already in
use somewhere. Does it make sense to have a RFE for better integration
between OVN and neutron-dynamic-routing?

Thanks
Francois




[1]: https://www.openvswitch.org/support/ovscon2020/slides/OVS-CONF-2020-OVN-WITH-DYNAMIC-ROUTING.pdf
[2]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html
[3]: https://ltomasbo.wordpress.com/2021/02/04/openstack-networking-with-bgp/
[4]: https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/triplo-bgp-frrouter.html
[5]: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/neutron-bgpvpn.yaml