Thanks, Rodolfo. I'll take a look at each of these after coffee and clarify my position (if needed). James On 3/2/20, 6:27 AM, "Rodolfo Alonso" <ralonsoh@redhat.com> wrote: CAUTION: This message originated externally, please use caution when clicking on links or opening attachments! Hello James: Just to make a quick summary of the status of the commented bugs/regressions: 1) https://bugs.launchpad.net/neutron/+bug/1810563: adding rules to security groups is slow That was addressed in https://review.opendev.org/#/c/633145/ and https://review.opendev.org/#/c/637407/, removing the O^2 check and using lazy loading. 2) https://bugzilla.redhat.com/show_bug.cgi?id=1788749: Neutron List networks API regression The last reply was marked as private. I've undone this and you can read now c#2. Testing with a similar scenario, I don't see any performance degradation between Queens and Train. 3) https://bugzilla.redhat.com/show_bug.cgi?id=1721273: Neutron API List Ports Performance regression That problem was solved in https://review.opendev.org/#/c/667981/ and https://review.opendev.org/#/c/667998/, by refactoring how the port QoS extension was reading and applying the QoS info in the port dict. 4) https://bugs.launchpad.net/neutron/+bug/1865223: regression for security group list between Newton and Rocky+ This is similar to https://bugs.launchpad.net/neutron/+bug/1863201. In this case, the regression was detected from R to S. The performance dropped from 3 secs to 110 secs (36x). That issue was addressed by https://review.opendev.org/#/c/708695/. But while 1865223 is talking about *SG list*, 1863201 is related to *SG rule list*. I would like to make this differentiation, because both retrieval commands are not related. In this bug (1863201), the performance degradation multiplies by x3 (N->Q) the initial time. This could be caused by the OVO integration (O->P: https://review.opendev.org/#/c/284738/). Instead of using the DB object now we make this call using the OVO object containing the DB register (something like a DB view). That's something I still need to check. Just to make a concretion: the patch 708695 improves the *SG rule* retrieval, not the SG list command. Another punctualization is that this patch will help in the case of having a balance between SG rules and SG. This patch will help to retrieve from the DB only those SG rules belonging to the project. If, as you state in https://bugs.launchpad.net/neutron/+bug/1865223/comments/4, most of those SG rules belong to the same project, there is little improvement there. As commented, I'm still looking at improving the SG OVO performance. Regards On Mon, 2020-03-02 at 03:03 +0000, Erik Olof Gunnar Andersson wrote: > When we went from Mitaka to Rocky in August last year and we saw an exponential increase in api > times for listing security group rules. > > I think I last commented on this bug https://bugs.launchpad.net/neutron/+bug/1810563, but I have > brought it up on a few other occasions as well. > Bug #1810563 “adding rules to security groups is slow” : Bugs : neutron Sometime between liberty > and pike, adding rules to SG's got slow, and slower with every rule added. Gerrit review with > fixes is incoming. You can repro with a vanilla devstack install on master, and this script: > #!/bin/bash OPENSTACK_TOKEN=$(openstack token issue | grep '| id' | awk '{print $4}') export > OPENSTACK_TOKEN CCN1=10.210.162.2 CCN3=10.210.162.10 export ENDPOINT=localhost make_rules() { > iter=$1 prefix=$2 file="$3" echo "generating rules" cat >$file <<EOF > {... bugs.launchpad.net > > > From: Slawek Kaplonski <skaplons@redhat.com> > Sent: Saturday, February 29, 2020 12:44 AM > To: James Denton <james.denton@rackspace.com> > Cc: openstack-discuss <openstack-discuss@lists.openstack.org> > Subject: Re: [neutron] security group list regression > > Hi, > > I just replied in Your bug report. Can You try to apply patch > https://urldefense.com/v3/__https://review.opendev.org/*/c/708695/__;Iw!!Ci6... > to see if that will help with this problem? > > > On 29 Feb 2020, at 02:41, James Denton <james.denton@rackspace.com> wrote: > > > > Hello all, > > > > We recently upgraded an environment from Newton -> Rocky, and have noticed a pretty severe > regression in the time it takes the API to return the list of security groups. This environment > has roughly 8,000+ security groups, and it takes nearly 75 seconds for the ‘openstack security > group list’ command to complete. I don’t have actual data from the same environment running > Newton, but was able to replicate this behavior with the following lab environments running a mix > of virtual and baremetal machines: > > > > Newton (VM) > > Rocky (BM) > > Stein (VM) > > Train (BM) > > > > Number of sec grps vs time in seconds: > > > > # Newton Rocky Stein Train > > 200 4.1 3.7 5.4 5.2 > > 500 5.3 7 11 9.4 > > 1000 7.2 12.4 19.2 16 > > 2000 9.2 24.2 35.3 30.7 > > 3000 12.1 36.5 52 44 > > 4000 16.1 47.2 73 58.9 > > 5000 18.4 55 90 69 > > > > As you can see (hopefully), the response time increased significantly between Newton and Rocky, > and has grown slightly ever since. We don't know, yet, if this behavior can be seen with other > 'list' commands or is limited to secgroups. We're currently verifying on some intermediate > releases to see where things went wonky. > > > > There are some similar recent reports out in the wild with little feedback: > > > > > https://urldefense.com/v3/__https://bugzilla.redhat.com/show_bug.cgi?id=1788... > > > > https://urldefense.com/v3/__https://bugzilla.redhat.com/show_bug.cgi?id=1721... > > > > > I opened a bug here, too: > > > > > https://urldefense.com/v3/__https://bugs.launchpad.net/neutron/*bug/1865223_... > > > > > Bottom line: Has anyone else experienced similar regressions in recent releases? If so, were you > able to address them with any sort of tuning? > > > > Thanks in advance, > > James > > > > — > Slawek Kaplonski > Senior software engineer > Red Hat > >