<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div>Hi Daniel,<br></div><div><br></div><div>Sure, I've opened <a href="https://review.opendev.org/#/c/752678/">https://review.opendev.org/#/c/752678/</a> so we can move the discussion there - I'll add tests next week.<br></div><div><br></div><div id="sig53546321"><div>--<br></div><div> Krzysztof Klimonda<br></div><div> kklimonda@syntaxhighlighted.com<br></div><div><br></div></div><div><br></div><div><br></div><div>On Fri, Sep 18, 2020, at 11:17, Daniel Alvarez Sanchez wrote:<br></div><blockquote type="cite" id="qt" style=""><div dir="ltr"><div dir="ltr">Hey folks, thanks for bringing this up!<br></div><div><br></div><div class="qt-gmail_quote"><div dir="ltr" class="qt-gmail_attr">On Fri, Sep 18, 2020 at 10:39 AM Krzysztof Klimonda <<a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a>> wrote:<br></div><blockquote class="qt-gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div>So just for testing I've applied this patch to our neutron-server:<br></div><div> <br></div><div> --8<--8<--8<--<br></div><div> diff --git a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br></div><div> index 23a841d7a1..41200786f1 100644<br></div><div> --- a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br></div><div> +++ b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py<br></div><div> @@ -1141,11 +1141,15 @@ class OVNClient(object):<br></div><div> enabled = router.get('admin_state_up')<br></div><div> lrouter_name = utils.ovn_name(router['id'])<br></div><div> added_gw_port = None<br></div><div> + options = {<br></div><div> + "always_learn_from_arp_request": "false",<br></div><div> + "dynamic_neigh_routers": "true"<br></div><div> + }<br></div><div> with self._nb_idl.transaction(check_error=True) as txn:<br></div><div> txn.add(self._nb_idl.create_lrouter(lrouter_name,<br></div><div> external_ids=external_ids,<br></div><div> enabled=enabled,<br></div><div> - options={}))<br></div><div> + options=options))<br></div><div> # TODO(lucasagomes): add_external_gateway is being only used<br></div><div> # by the ovn_db_sync.py script, remove it after the database<br></div><div> # synchronization work<br></div><div> --8<--8<--8<--<br></div><div> <br></div></blockquote><div> <br></div><div>Do you want to propose a formal patch with this change and some test? I think we may want this in.<br></div><div>The question is if we want to make it configurable and, if so, how to expose it to an admin...<br></div><div><br></div><div>When enabling this, we're going to have ARP requests between the logical routers and, depending on the topology, this can create other scaling issues due to MAC_Binding entries in the Southbound database. Also, these entries do not age out [0] so we might end up with a very large amount of rows here.<br></div><div><br></div><div>This is probably an unlikely pattern in OpenStack where usually projects connect their routers to a localnet Logical Switch but routers across projects are not interconnected. Hence, the amount of E/W traffic is not a full mesh and, probably, even in the worst case it's going to be still better than what we have right now in terms of scaling.<br></div><div><br></div><div>On the ARP traffic side, it should not be a big deal, they'll be sent between routers when needed and, since the entries do not age out, they're not going to be sent again.<br></div><div><br></div><div>Again, thanks for bringing this up! I'd +1 this change :) <br></div><div> <br></div><div>[0] <a href="https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg05917.html" target="_blank">https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg05917.html</a><br></div><div><br></div><div><br></div><blockquote class="qt-gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div>and also executed that for each logical router in OVN:<br></div><div> <br></div><div> # ovn-nbctl set Logical_Router $router options=dynamic_neigh_routers=true,always_learn_from_arp_request=false<br></div><div> <br></div><div> This had a huge impact on both a number of logical flows and a number of ovs flows on chassis nodes:<br></div><div> <br></div><div> --8<--8<--8<--<br></div><div> # cat lflows-new.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1 |sort | uniq -c |sort -n | tail -10<br></div><div> 2170 ls_out_port_sec_l2<br></div><div> 2172 lr_in_learn_neighbor<br></div><div> 2666 lr_in_admission<br></div><div> 2690 ls_in_port_sec_l2<br></div><div> 3190 lr_in_ip_routing<br></div><div> 4276 lr_in_lookup_neighbor<br></div><div> 4873 lr_in_arp_resolve<br></div><div> 5864 ls_in_arp_rsp<br></div><div> 5873 ls_in_l2_lkup<br></div><div> 14343 lr_in_ip_input<br></div><div> # ovn-sbctl --timeout=120 lflow-list > lflows-new.txt<br></div><div> --8<--8<--8<--<br></div><div> <br></div><div> (and this is even more routers than before - 500 vs 400). I'll have to read what impact do those options have on ARP activity though.<br></div><div> <br></div><div> -- <br></div><div> Krzysztof Klimonda<br></div><div> <a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br></div><div> <br></div><div> On Thu, Sep 17, 2020, at 21:14, Krzysztof Klimonda wrote:<br></div><div> > Hi Tony,<br></div><div> > <br></div><div> > Indeed I forgot to mention that all routers are using the same external <br></div><div> > network (and subnet) for the external gateway.<br></div><div> > <br></div><div> > Creating separate external networks per router wouldn't really work for <br></div><div> > us, and I'm not even quite sure what the setup would look like in that <br></div><div> > case. <br></div><div> > <br></div><div> > -- <br></div><div> > Krzysztof Klimonda<br></div><div> > <a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br></div><div> > <br></div><div> > On Thu, Sep 17, 2020, at 20:31, Tony Liu wrote:<br></div><div> > > I am trying to reach 5000. The problem I hit is that northd is<br></div><div> > > stuck in translating from NB to SB when connect router to external<br></div><div> > > network.<br></div><div> > > <br></div><div> > > I assume all your 400 routers connect to the same subnet in that<br></div><div> > > external network. I am trying another approach where one subnet<br></div><div> > > is created for each router in external network. That may help to<br></div><div> > > reduce the ARP flow?<br></div><div> > > <br></div><div> > > Thanks!<br></div><div> > > Tony<br></div><div> > > > -----Original Message-----<br></div><div> > > > From: Krzysztof Klimonda <<a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a>><br></div><div> > > > Sent: Thursday, September 17, 2020 8:57 AM<br></div><div> > > > To: <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br></div><div> > > > Subject: [neutron][ovn] Logical flow scaling (flow explosion in<br></div><div> > > > lr_in_arp_resolve)<br></div><div> > > > <br></div><div> > > > Hi,<br></div><div> > > > <br></div><div> > > > We're running some tests of ussuri deployment with ovn ML2 driver and<br></div><div> > > > seeing some worrying numbers of logical flows generated for our test<br></div><div> > > > deployment.<br></div><div> > > > <br></div><div> > > > As a test, we create 400 routes, 400 private networks and connect each<br></div><div> > > > network to its own routers. We also connect each router to an external<br></div><div> > > > network. After doing that a dump of logical flows shows almost 800k<br></div><div> > > > logical flows, most of them in lr_in_arp_resolve table:<br></div><div> > > > <br></div><div> > > > --8<--8<--8<--<br></div><div> > > > # cat lflows.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1<br></div><div> > > > |sort | uniq -c |sort -n | tail -10<br></div><div> > > > 3264 lr_in_learn_neighbor<br></div><div> > > > 3386 ls_out_port_sec_l2<br></div><div> > > > 4112 lr_in_admission<br></div><div> > > > 4202 ls_in_port_sec_l2<br></div><div> > > > 4898 lr_in_lookup_neighbor<br></div><div> > > > 4900 lr_in_ip_routing<br></div><div> > > > 9144 ls_in_l2_lkup<br></div><div> > > > 9160 ls_in_arp_rsp<br></div><div> > > > 22136 lr_in_ip_input<br></div><div> > > > 671656 lr_in_arp_resolve<br></div><div> > > > #<br></div><div> > > > --8<--8<--8<--<br></div><div> > > > <br></div><div> > > > ovn: 20.06.2 + patch for SNAT IP ARP reply issue<br></div><div> > > > openvswitch: 2.13.0<br></div><div> > > > neutron: 16.1.0<br></div><div> > > > <br></div><div> > > > I've seen some discussion about similar issue at OVS mailing lists:<br></div><div> > > > <a href="https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07014.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07014.html</a> -<br></div><div> > > > is this relevant to neutron, and not just kubernetes?<br></div><div> > > > <br></div><div> > > > --<br></div><div> > > > Krzysztof Klimonda<br></div><div> > > > <a href="mailto:kklimonda@syntaxhighlighted.com" target="_blank">kklimonda@syntaxhighlighted.com</a><br></div><div> > > <br></div><div> > ><br></div><div> > <br></div><div> ><br></div><div> <br></div></blockquote></div></div></blockquote><div><br></div></body></html>