(yes we are using distributed fips) Well we don't want stretched
VLANs. However... if we followed the doc we would end up with 3
controllers in the same rack which would not be resilient. Since we
have just 3 controllers plugged on specific, identified ports, we
afford to have a stretched VLAN on these few ports only. For the
provisioning network, I am taking a shortcut since this network should
basically only be needed once in a while for stack upgrades and
nothing interesting (like mac addresses moving) happens there. The
data plane traffic, that needs scalability and resiliency, is not
going through these VLANs. I think stretched VLANs on leaf spine
networks are forbidden in general for these reasons (large L2
networks? STP reducing the bandwidth? broadcast storm? larger failure
domain? I don't know specifically, I would need help from a network
engineer to explain the reason).

 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/spine_leaf_networking/index


On Tue, 12 Oct 2021 at 18:04, Tony Liu <tonyliu0592@hotmail.com> wrote:
>
> I wonder, since there is already VXLAN based L2 across multiple racks
> (which means you are not looking for pure L3 solution),
> while keep tenant network multi-subnet on L3 for EW traffic,
> why not have external network also on L2 and stretched on multiple racks
> for NS traffic, assuming you are using distributed FIP?
>
> Thanks!
> Tony
> ________________________________________
> From: Francois <rigault.francois@gmail.com>
> Sent: October 12, 2021 07:03 AM
> To: openstack-discuss
> Subject: [neutron] OVN and dynamic routing
>
> Hello Neutron!
> I am looking into running stacks with OVN on a leaf-spine network, and
> have some floating IPs routed between racks.
>
> Basically each rack is assigned its own set of subnets.
> Some VLANs are stretched across all racks: the provisioning VLAN used
> by tripleo to deploy the stack, and the VLANs for the controllers API
> IPs. However, each tenant subnet is local to a rack: for example each
> OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its
> own rack. Traffic between 2 racks is sent to a spine, and leaves and
> spines run some eVPN-like thing: each pair of ToR is a vtep, traffic
> is encapsulated as VXLAN, and routes between vteps are exchanged with
> BGP.
>
> I am looking into supporting floating IPs in there: I expect floating
> IPs to be able to move between racks, as such I am looking into
> publishing the route for a FIP towards an hypervisor, through BGP.
> Each fip is a /32 subnet with an hypervisor tenant's IP as next hop.
>
> It seems there are several ideas to achieve this (it was discussed
> [before][1] in ovs conference)
> - using [neutron-dynamic-routing][2] - that seems to have some gaps
> for OVN. It uses os-ken to talk to switches and exchange routes
> - using [OVN BGP agent][3] that uses FRR, it seems there is a related
> [RFE][4] for integration in tripleo
>
> There is btw also a [BGPVPN][5] project (it does not match my usecase
> as far as I tried to understand it) that also has some code that talks
> BGP to switches, already integrated in tripleo.
>
> For my tests, I was able to use the neutron-dynamic-routing project
> (almost) as documented, with a few changes:
> - for traffic going from VMs to outside the stack, the hypervisor was
> trying to resolve the "gateway of fIPs" with ARP request which does
> not make any sense. I created a dummy port with the mac address of the
> virtual router of the switches:
> ```
> $ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml
> - Fixed IP Addresses:
> - ip_address: 10.64.254.1
> subnet_id: 8f37
> ID: 4028
> MAC Address: 00:1c:73:00:00:11
> Name: lagw
> Status: DOWN
> ```
> this prevent the hypervisor to send ARP requests to a non existent gateway
> - for traffic coming back, we start the neutron-bgp-dragent agent on
> the controllers. We create the right bgp speaker, peers, etc.
> - neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it
> selects fips and join with ports owned by a "floatingip_agent_gateway"
> which does not exist on OVN. We can define ourselves some ports so
> that the dragent is able to find the tenant IP of a host:
> ```
> openstack port create --network provider --device-owner
> network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip
> ip-address=10.64.245.102 ag2
> ```
> - when creating a floating IP and assigning a port to it, Neutron
> reads changes from OVN SB and fills the binding information into the
> port:
> ```
> $ openstack port show -c binding_host_id `openstack floating ip show
> 10.64.254.177 -f value -c port_id`
> +-----------------+----------------------------------------+
> | Field | Value |
> +-----------------+----------------------------------------+
> | binding_host_id | cpu35d.cloud |
> +-----------------+----------------------------------------+
> ```
> this allows the dragent to publish the route for the fip
> ```
> $ openstack bgp speaker list advertised routes bgpspeaker
> +------------------+---------------+
> | Destination | Nexthop |
> +------------------+---------------+
> | 10.64.254.177/32 | 10.64.245.102 |
> +------------------+---------------+
> ```
> - traffic reaches the hypervisor but (for reason I don't understand) I
> had to add a rule
> ```
> $ ip rule
> 0: from all lookup local
> 32765: from all iif vlan1234 lookup ovn
> 32766: from all lookup main
> 32767: from all lookup default
> $ ip route show table ovn
> 10.64.254.177 dev vlan1234 scope link
> ```
> so that the traffic coming for the fip is not immediately discarded by
> the hypervisor (it's not an ideal solution but it is a workaround that
> makes my one fIP work!)
>
> So all in all it seems it would be possible to use the
> neutron-dynamic-routing agent, with some minor modifications (eg: to
> also publish the fip of the OVN L3 gateway router).
>
> I am wondering whether I have overlooked anything, and if such kind of
> deployment (OVN + neutron dynamic routing or similar) is already in
> use somewhere. Does it make sense to have a RFE for better integration
> between OVN and neutron-dynamic-routing?
>
> Thanks
> Francois
>
>
>
>
> [1]: https://www.openvswitch.org/support/ovscon2020/slides/OVS-CONF-2020-OVN-WITH-DYNAMIC-ROUTING.pdf
> [2]: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html
> [3]: https://ltomasbo.wordpress.com/2021/02/04/openstack-networking-with-bgp/
> [4]: https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/triplo-bgp-frrouter.html
> [5]: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/neutron-bgpvpn.yaml
>