Open Stack

Hi Daniel,

Sure, I've opened https://review.opendev.org/#/c/752678/ so we can move the discussion there - I'll add tests next week.

--
  Krzysztof Klimonda
  kklimonda at syntaxhighlighted.com



On Fri, Sep 18, 2020, at 11:17, Daniel Alvarez Sanchez wrote:
> Hey folks, thanks for bringing this up!
> 
> On Fri, Sep 18, 2020 at 10:39 AM Krzysztof Klimonda <kklimonda at syntaxhighlighted.com> wrote:
>> So just for testing I've applied this patch to our neutron-server:
>> 
>> --8<--8<--8<--
>> diff --git a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
>> index 23a841d7a1..41200786f1 100644
>> --- a/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
>> +++ b/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py
>> @@ -1141,11 +1141,15 @@ class OVNClient(object):
>>          enabled = router.get('admin_state_up')
>>          lrouter_name = utils.ovn_name(router['id'])
>>          added_gw_port = None
>> +        options = {
>> +          "always_learn_from_arp_request": "false",
>> +          "dynamic_neigh_routers": "true"
>> +        }
>>          with self._nb_idl.transaction(check_error=True) as txn:
>>              txn.add(self._nb_idl.create_lrouter(lrouter_name,
>>                                                  external_ids=external_ids,
>>                                                  enabled=enabled,
>> -                                                options={}))
>> +                                                options=options))
>>              # TODO(lucasagomes): add_external_gateway is being only used
>>              # by the ovn_db_sync.py script, remove it after the database
>>              # synchronization work
>> --8<--8<--8<--
>> 
>  
> Do you want to propose a formal patch with this change and some test? I think we may want this in.
> The question is if we want to make it configurable and, if so, how to expose it to an admin...
> 
> When enabling this, we're going to have ARP requests between the logical routers and, depending on the topology, this can create other scaling issues due to MAC_Binding entries in the Southbound database. Also, these entries do not age out [0] so we might end up with a very large amount of rows here.
> 
> This is probably an unlikely pattern in OpenStack where usually projects connect their routers to a localnet Logical Switch but routers across projects are not interconnected. Hence, the amount of E/W traffic is not a full mesh and, probably, even in the worst case it's going to be still better than what we have right now in terms of scaling.
> 
> On the ARP traffic side, it should not be a big deal, they'll be sent between routers when needed and, since the entries do not age out, they're not going to be sent again.
> 
> Again, thanks for bringing this up! I'd +1 this change :) 
>  
> [0] https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg05917.html
> 
> 
>> and also executed that for each logical router in OVN:
>> 
>> # ovn-nbctl set Logical_Router $router options=dynamic_neigh_routers=true,always_learn_from_arp_request=false
>> 
>> This had a huge impact on both a number of logical flows and a number of ovs flows on chassis nodes:
>> 
>> --8<--8<--8<--
>> # cat lflows-new.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1 |sort | uniq -c |sort -n | tail -10
>>    2170 ls_out_port_sec_l2
>>    2172 lr_in_learn_neighbor
>>    2666 lr_in_admission
>>    2690 ls_in_port_sec_l2
>>    3190 lr_in_ip_routing
>>    4276 lr_in_lookup_neighbor
>>    4873 lr_in_arp_resolve
>>    5864 ls_in_arp_rsp
>>    5873 ls_in_l2_lkup
>>   14343 lr_in_ip_input
>> # ovn-sbctl --timeout=120 lflow-list > lflows-new.txt
>> --8<--8<--8<--
>> 
>> (and this is even more routers than before - 500 vs 400). I'll have to read what impact do those options have on ARP activity though.
>> 
>> -- 
>>   Krzysztof Klimonda
>>   kklimonda at syntaxhighlighted.com
>> 
>> On Thu, Sep 17, 2020, at 21:14, Krzysztof Klimonda wrote:
>> > Hi Tony,
>> > 
>> > Indeed I forgot to mention that all routers are using the same external 
>> > network (and subnet) for the external gateway.
>> > 
>> > Creating separate external networks per router wouldn't really work for 
>> > us, and I'm not even quite sure what the setup would look like in that 
>> > case. 
>> > 
>> > -- 
>> >   Krzysztof Klimonda
>> >   kklimonda at syntaxhighlighted.com
>> > 
>> > On Thu, Sep 17, 2020, at 20:31, Tony Liu wrote:
>> > > I am trying to reach 5000. The problem I hit is that northd is
>> > > stuck in translating from NB to SB when connect router to external
>> > > network.
>> > > 
>> > > I assume all your 400 routers connect to the same subnet in that
>> > > external network. I am trying another approach where one subnet
>> > > is created for each router in external network. That may help to
>> > > reduce the ARP flow?
>> > > 
>> > > Thanks!
>> > > Tony
>> > > > -----Original Message-----
>> > > > From: Krzysztof Klimonda <kklimonda at syntaxhighlighted.com>
>> > > > Sent: Thursday, September 17, 2020 8:57 AM
>> > > > To: openstack-discuss at lists.openstack.org
>> > > > Subject: [neutron][ovn] Logical flow scaling (flow explosion in
>> > > > lr_in_arp_resolve)
>> > > > 
>> > > > Hi,
>> > > > 
>> > > > We're running some tests of ussuri deployment with ovn ML2 driver and
>> > > > seeing some worrying numbers of logical flows generated for our test
>> > > > deployment.
>> > > > 
>> > > > As a test, we create 400 routes, 400 private networks and connect each
>> > > > network to its own routers. We also connect each router to an external
>> > > > network. After doing that a dump of logical flows shows almost 800k
>> > > > logical flows, most of them in lr_in_arp_resolve table:
>> > > > 
>> > > > --8<--8<--8<--
>> > > > # cat lflows.txt |grep -v Datapath |cut -d'(' -f 2 | cut -d ')' -f1
>> > > > |sort | uniq -c |sort -n | tail -10
>> > > >    3264 lr_in_learn_neighbor
>> > > >    3386 ls_out_port_sec_l2
>> > > >    4112 lr_in_admission
>> > > >    4202 ls_in_port_sec_l2
>> > > >    4898 lr_in_lookup_neighbor
>> > > >    4900 lr_in_ip_routing
>> > > >    9144 ls_in_l2_lkup
>> > > >    9160 ls_in_arp_rsp
>> > > >   22136 lr_in_ip_input
>> > > >  671656 lr_in_arp_resolve
>> > > > #
>> > > > --8<--8<--8<--
>> > > > 
>> > > > ovn: 20.06.2 + patch for SNAT IP ARP reply issue
>> > > > openvswitch: 2.13.0
>> > > > neutron: 16.1.0
>> > > > 
>> > > > I've seen some discussion about similar issue at OVS mailing lists:
>> > > > https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07014.html -
>> > > > is this relevant to neutron, and not just kubernetes?
>> > > > 
>> > > > --
>> > > >   Krzysztof Klimonda
>> > > >   kklimonda at syntaxhighlighted.com
>> > > 
>> > >
>> > 
>> >
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200918/e3bab4a8/attachment-0001.html>