<div dir="ltr"><div dir="ltr">Hey folks, thanks for bringing this up!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 18, 2020 at 10:39 AM Krzysztof Klimonda <<a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">So just for testing I've applied this patch to our neutron-server:<br>
<br>
--8<--8<--8<--<br>
diff --git a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br>
index 23a841d7a1..41200786f1 100644<br>
--- a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br>
+++ b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br>
@@ -1141,11 +1141,15 @@ class OVNClient(object):<br>
enabled = router.get('admin_state_up')<br>
lrouter_name = utils.ovn_name(router['id'])<br>
added_gw_port = None<br>
+ options = {<br>
+ "always_learn_from_arp_request": "false",<br>
+ "dynamic_neigh_routers": "true"<br>
+ }<br>
with self._nb_idl.transaction(check_error=True) as txn:<br>
txn.add(self._nb_idl.create_lrouter(lrouter_name,<br>
external_ids=external_ids,<br>
enabled=enabled,<br>
- options={}))<br>
+ options=options))<br>
# TODO(lucasagomes): add_external_gateway is being only used<br>
# by the ovn_db_sync.py script, remove it after the database<br>
# synchronization work<br>
--8<--8<--8<--<br>
<br></blockquote><div> </div><div>Do you want to propose a formal patch with this change and some test? I think we may want this in.</div><div>The question is if we want to make it configurable and, if so, how to expose it to an admin...</div><div><br></div><div>When enabling this, we're going to have ARP requests between the logical routers and, depending on the topology, this can create other scaling issues due to MAC_Binding entries in the Southbound database. Also, these entries do not age out [0] so we might end up with a very large amount of rows here.</div><div><br></div><div>This is probably an unlikely pattern in OpenStack where usually projects connect their routers to a localnet Logical Switch but routers across projects are not interconnected. Hence, the amount of E/W traffic is not a full mesh and, probably, even in the worst case it's going to be still better than what we have right now in terms of scaling.</div><div><br></div><div>On the ARP traffic side, it should not be a big deal, they'll be sent between routers when needed and, since the entries do not age out, they're not going to be sent again.</div><div><br></div><div>Again, thanks for bringing this up! I'd +1 this change :) </div><div> </div><div>[0] <a href="https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg05917.html" target="_blank">https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg05917.html</a></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
and also executed that for each logical router in OVN:<br>
<br>
# ovn-nbctl set Logical_Router $router options=dynamic_neigh_routers=true,always_learn_from_arp_request=false<br>
<br>
This had a huge impact on both a number of logical flows and a number of ovs flows on chassis nodes:<br>
<br>
--8<--8<--8<--<br>
# cat lflows-new.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1 |sort | uniq -c |sort -n | tail -10<br>
2170 ls_out_port_sec_l2<br>
2172 lr_in_learn_neighbor<br>
2666 lr_in_admission<br>
2690 ls_in_port_sec_l2<br>
3190 lr_in_ip_routing<br>
4276 lr_in_lookup_neighbor<br>
4873 lr_in_arp_resolve<br>
5864 ls_in_arp_rsp<br>
5873 ls_in_l2_lkup<br>
14343 lr_in_ip_input<br>
# ovn-sbctl --timeout=120 lflow-list > lflows-new.txt<br>
--8<--8<--8<--<br>
<br>
(and this is even more routers than before - 500 vs 400). I'll have to read what impact do those options have on ARP activity though.<br>
<br>
-- <br>
Krzysztof Klimonda<br>
<a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br>
<br>
On Thu, Sep 17, 2020, at 21:14, Krzysztof Klimonda wrote:<br>
> Hi Tony,<br>
> <br>
> Indeed I forgot to mention that all routers are using the same external <br>
> network (and subnet) for the external gateway.<br>
> <br>
> Creating separate external networks per router wouldn't really work for <br>
> us, and I'm not even quite sure what the setup would look like in that <br>
> case. <br>
> <br>
> -- <br>
> Krzysztof Klimonda<br>
> <a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br>
> <br>
> On Thu, Sep 17, 2020, at 20:31, Tony Liu wrote:<br>
> > I am trying to reach 5000. The problem I hit is that northd is<br>
> > stuck in translating from NB to SB when connect router to external<br>
> > network.<br>
> > <br>
> > I assume all your 400 routers connect to the same subnet in that<br>
> > external network. I am trying another approach where one subnet<br>
> > is created for each router in external network. That may help to<br>
> > reduce the ARP flow?<br>
> > <br>
> > Thanks!<br>
> > Tony<br>
> > > -----Original Message-----<br>
> > > From: Krzysztof Klimonda <<a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a>><br>
> > > Sent: Thursday, September 17, 2020 8:57 AM<br>
> > > To: <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
> > > Subject: [neutron][ovn] Logical flow scaling (flow explosion in<br>
> > > lr_in_arp_resolve)<br>
> > > <br>
> > > Hi,<br>
> > > <br>
> > > We're running some tests of ussuri deployment with ovn ML2 driver and<br>
> > > seeing some worrying numbers of logical flows generated for our test<br>
> > > deployment.<br>
> > > <br>
> > > As a test, we create 400 routes, 400 private networks and connect each<br>
> > > network to its own routers. We also connect each router to an external<br>
> > > network. After doing that a dump of logical flows shows almost 800k<br>
> > > logical flows, most of them in lr_in_arp_resolve table:<br>
> > > <br>
> > > --8<--8<--8<--<br>
> > > # cat lflows.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1<br>
> > > |sort | uniq -c |sort -n | tail -10<br>
> > > 3264 lr_in_learn_neighbor<br>
> > > 3386 ls_out_port_sec_l2<br>
> > > 4112 lr_in_admission<br>
> > > 4202 ls_in_port_sec_l2<br>
> > > 4898 lr_in_lookup_neighbor<br>
> > > 4900 lr_in_ip_routing<br>
> > > 9144 ls_in_l2_lkup<br>
> > > 9160 ls_in_arp_rsp<br>
> > > 22136 lr_in_ip_input<br>
> > > 671656 lr_in_arp_resolve<br>
> > > #<br>
> > > --8<--8<--8<--<br>
> > > <br>
> > > ovn: 20.06.2 + patch for SNAT IP ARP reply issue<br>
> > > openvswitch: 2.13.0<br>
> > > neutron: 16.1.0<br>
> > > <br>
> > > I've seen some discussion about similar issue at OVS mailing lists:<br>
> > > <a href="https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07014.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07014.html</a> -<br>
> > > is this relevant to neutron, and not just kubernetes?<br>
> > > <br>
> > > --<br>
> > > Krzysztof Klimonda<br>
> > > <a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br>
> > <br>
> ><br>
> <br>
><br>
<br>
</blockquote></div></div>