So just for testing I've applied this patch to our neutron-server:
--8<--8<--8<--
diff --git a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
index 23a841d7a1..41200786f1 100644
--- a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
+++ b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
@@ -1141,11 +1141,15 @@ class OVNClient(object):
enabled = router.get('admin_state_up')
lrouter_name = utils.ovn_name(router['id'])
added_gw_port = None
+ options = {
+ "always_learn_from_arp_request": "false",
+ "dynamic_neigh_routers": "true"
+ }
with self._nb_idl.transaction(check_error=True) as txn:
txn.add(self._nb_idl.create_lrouter(lrouter_name,
external_ids=external_ids,
enabled=enabled,
- options={}))
+ options=options))
# TODO(lucasagomes): add_external_gateway is being only used
# by the ovn_db_sync.py script, remove it after the database
# synchronization work
--8<--8<--8<--
Do you want to propose a formal patch with this change and some test? I think we may want this in.
The question is if we want to make it configurable and, if so, how to expose it to an admin...
When enabling this, we're going to have ARP requests between the logical routers and, depending on the topology, this can create other scaling issues due to MAC_Binding entries in the Southbound database. Also, these entries do not age out [0] so we might end up with a very large amount of rows here.
This is probably an unlikely pattern in OpenStack where usually projects connect their routers to a localnet Logical Switch but routers across projects are not interconnected. Hence, the amount of E/W traffic is not a full mesh and, probably, even in the worst case it's going to be still better than what we have right now in terms of scaling.
On the ARP traffic side, it should not be a big deal, they'll be sent between routers when needed and, since the entries do not age out, they're not going to be sent again.
Again, thanks for bringing this up! I'd +1 this change :)
and also executed that for each logical router in OVN:
# ovn-nbctl set Logical_Router $router options=dynamic_neigh_routers=true,always_learn_from_arp_request=false
This had a huge impact on both a number of logical flows and a number of ovs flows on chassis nodes:
--8<--8<--8<--
# cat lflows-new.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1 |sort | uniq -c |sort -n | tail -10
2170 ls_out_port_sec_l2
2172 lr_in_learn_neighbor
2666 lr_in_admission
2690 ls_in_port_sec_l2
3190 lr_in_ip_routing
4276 lr_in_lookup_neighbor
4873 lr_in_arp_resolve
5864 ls_in_arp_rsp
5873 ls_in_l2_lkup
14343 lr_in_ip_input
# ovn-sbctl --timeout=120 lflow-list > lflows-new.txt
--8<--8<--8<--
(and this is even more routers than before - 500 vs 400). I'll have to read what impact do those options have on ARP activity though.
--
Krzysztof Klimonda
On Thu, Sep 17, 2020, at 21:14, Krzysztof Klimonda wrote:
> Hi Tony,
>
> Indeed I forgot to mention that all routers are using the same external
> network (and subnet) for the external gateway.
>
> Creating separate external networks per router wouldn't really work for
> us, and I'm not even quite sure what the setup would look like in that
> case.
>
> --
> Krzysztof Klimonda
>
> On Thu, Sep 17, 2020, at 20:31, Tony Liu wrote:
> > I am trying to reach 5000. The problem I hit is that northd is
> > stuck in translating from NB to SB when connect router to external
> > network.
> >
> > I assume all your 400 routers connect to the same subnet in that
> > external network. I am trying another approach where one subnet
> > is created for each router in external network. That may help to
> > reduce the ARP flow?
> >
> > Thanks!
> > Tony
> > > -----Original Message-----
> > > Sent: Thursday, September 17, 2020 8:57 AM
> > > Subject: [neutron][ovn] Logical flow scaling (flow explosion in
> > > lr_in_arp_resolve)
> > >
> > > Hi,
> > >
> > > We're running some tests of ussuri deployment with ovn ML2 driver and
> > > seeing some worrying numbers of logical flows generated for our test
> > > deployment.
> > >
> > > As a test, we create 400 routes, 400 private networks and connect each
> > > network to its own routers. We also connect each router to an external
> > > network. After doing that a dump of logical flows shows almost 800k
> > > logical flows, most of them in lr_in_arp_resolve table:
> > >
> > > --8<--8<--8<--
> > > # cat lflows.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1
> > > |sort | uniq -c |sort -n | tail -10
> > > 3264 lr_in_learn_neighbor
> > > 3386 ls_out_port_sec_l2
> > > 4112 lr_in_admission
> > > 4202 ls_in_port_sec_l2
> > > 4898 lr_in_lookup_neighbor
> > > 4900 lr_in_ip_routing
> > > 9144 ls_in_l2_lkup
> > > 9160 ls_in_arp_rsp
> > > 22136 lr_in_ip_input
> > > 671656 lr_in_arp_resolve
> > > #
> > > --8<--8<--8<--
> > >
> > > ovn: 20.06.2 + patch for SNAT IP ARP reply issue
> > > openvswitch: 2.13.0
> > > neutron: 16.1.0
> > >
> > > I've seen some discussion about similar issue at OVS mailing lists:
> > > is this relevant to neutron, and not just kubernetes?
> > >
> > > --
> > > Krzysztof Klimonda
> >
> >
>
>