[neutron] security group list regression

Rodolfo Alonso ralonsoh at redhat.com
Mon Mar 2 11:26:33 UTC 2020


Hello James:

Just to make a quick summary of the status of the commented bugs/regressions:

1) https://bugs.launchpad.net/neutron/+bug/1810563: adding rules to security groups is slow
That was addressed in https://review.opendev.org/#/c/633145/ and 
https://review.opendev.org/#/c/637407/, removing the O^2 check and using lazy loading.


2) https://bugzilla.redhat.com/show_bug.cgi?id=1788749: Neutron List networks API regression
The last reply was marked as private. I've undone this and you can read now c#2. Testing with a
similar scenario, I don't see any performance degradation between Queens and Train.


3) https://bugzilla.redhat.com/show_bug.cgi?id=1721273: Neutron API List Ports Performance
regression
That problem was solved in https://review.opendev.org/#/c/667981/ and 
https://review.opendev.org/#/c/667998/, by refactoring how the port QoS extension was reading and
applying the QoS info in the port dict.


4) https://bugs.launchpad.net/neutron/+bug/1865223: regression for security group list between
Newton and Rocky+

This is similar to https://bugs.launchpad.net/neutron/+bug/1863201. In this case, the regression was
detected from R to S. The performance dropped from 3 secs to 110 secs (36x). That issue was
addressed by https://review.opendev.org/#/c/708695/.

But while 1865223 is talking about *SG list*, 1863201 is related to *SG rule list*. I would like to
make this differentiation, because both retrieval commands are not related.

In this bug (1863201), the performance degradation multiplies by x3 (N->Q) the initial time. This
could be caused by the OVO integration (O->P: https://review.opendev.org/#/c/284738/). Instead of
using the DB object now we make this call using the OVO object containing the DB register (something
like a DB view). That's something I still need to check.

Just to make a concretion: the patch 708695 improves the *SG rule* retrieval, not the SG list
command. Another punctualization is that this patch will help in the case of having a balance
between SG rules and SG. This patch will help to retrieve from the DB only those SG rules belonging
to the project. If, as you state in https://bugs.launchpad.net/neutron/+bug/1865223/comments/4, most
of those SG rules belong to the same project, there is little improvement there.

As commented, I'm still looking at improving the SG OVO performance.

Regards

	
On Mon, 2020-03-02 at 03:03 +0000, Erik Olof Gunnar Andersson wrote:
> When we went from Mitaka to Rocky in August last year and we saw an exponential increase in api
> times for listing security group rules.
> 
> I think I last commented on this bug https://bugs.launchpad.net/neutron/+bug/1810563, but I have
> brought it up on a few other occasions as well.
>  Bug #1810563 “adding rules to security groups is slow” : Bugs : neutron Sometime between liberty
> and pike, adding rules to SG's got slow, and slower with every rule added. Gerrit review with
> fixes is incoming. You can repro with a vanilla devstack install on master, and this script:
> #!/bin/bash OPENSTACK_TOKEN=$(openstack token issue | grep '| id' | awk '{print $4}') export
> OPENSTACK_TOKEN CCN1=10.210.162.2 CCN3=10.210.162.10 export ENDPOINT=localhost make_rules() {
> iter=$1 prefix=$2 file="$3" echo "generating rules" cat >$file <<EOF
> {... bugs.launchpad.net
> 
> 
> From: Slawek Kaplonski <skaplons at redhat.com>
> Sent: Saturday, February 29, 2020 12:44 AM
> To: James Denton <james.denton at rackspace.com>
> Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
> Subject: Re: [neutron] security group list regression
>  
> Hi,
> 
> I just replied in Your bug report. Can You try to apply patch 
> https://urldefense.com/v3/__https://review.opendev.org/*/c/708695/__;Iw!!Ci6f514n9QsL8ck!2GsBjp6V_V3EzrzAbWgNfsURfCm2tZmlUaw2J6OxFwJZUCV71lSP1b9jg8Ul-OlUqQ$
>   to see if that will help with this problem?
> 
> > On 29 Feb 2020, at 02:41, James Denton <james.denton at rackspace.com> wrote:
> > 
> > Hello all,
> > 
> > We recently upgraded an environment from Newton -> Rocky, and have noticed a pretty severe
> regression in the time it takes the API to return the list of security groups. This environment
> has roughly 8,000+ security groups, and it takes nearly 75 seconds for the ‘openstack security
> group list’ command to complete. I don’t have actual data from the same environment running
> Newton, but was able to replicate this behavior with the following lab environments running a mix
> of virtual and baremetal machines:
> > 
> > Newton (VM)
> > Rocky (BM)
> > Stein (VM)
> > Train (BM)
> >        
> > Number of sec grps vs time in seconds:
> >                
> > #     Newton Rocky Stein  Train
> > 200   4.1     3.7     5.4     5.2
> > 500   5.3     7       11      9.4
> > 1000  7.2     12.4    19.2    16
> > 2000  9.2     24.2    35.3    30.7
> > 3000  12.1    36.5    52      44
> > 4000  16.1    47.2    73      58.9
> > 5000  18.4    55      90      69
> > 
> > As you can see (hopefully), the response time increased significantly between Newton and Rocky,
> and has grown slightly ever since. We don't know, yet, if this behavior can be seen with other
> 'list' commands or is limited to secgroups. We're currently verifying on some intermediate
> releases to see where things went wonky.
> > 
> > There are some similar recent reports out in the wild with little feedback:
> > 
> > 
> https://urldefense.com/v3/__https://bugzilla.redhat.com/show_bug.cgi?id=1788749__;!!Ci6f514n9QsL8ck!2GsBjp6V_V3EzrzAbWgNfsURfCm2tZmlUaw2J6OxFwJZUCV71lSP1b9jg8Vx5jGlrA$
>  
> > 
> https://urldefense.com/v3/__https://bugzilla.redhat.com/show_bug.cgi?id=1721273__;!!Ci6f514n9QsL8ck!2GsBjp6V_V3EzrzAbWgNfsURfCm2tZmlUaw2J6OxFwJZUCV71lSP1b9jg8U9NbN_LA$
>  
> > 
> > I opened a bug here, too:
> > 
> > 
> https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1865223__;Kw!!Ci6f514n9QsL8ck!2GsBjp6V_V3EzrzAbWgNfsURfCm2tZmlUaw2J6OxFwJZUCV71lSP1b9jg8UtMQ2-Dw$
>  
> > 
> > Bottom line: Has anyone else experienced similar regressions in recent releases? If so, were you
> able to address them with any sort of tuning?
> > 
> > Thanks in advance,
> > James
> > 
> 
>> Slawek Kaplonski
> Senior software engineer
> Red Hat
> 
> 




More information about the openstack-discuss mailing list