[neutron] Secure RBAC api performance regression
Hi, In Neutron we are merging more and more patches related to the secure rbac policies [1]. Recently we noticed that our neutron-rally-task job is failing very often [2]. I did some tests and here is what were my results: * on latest neutron, with about 12 patches related to secure rbac merged - list of 4k ports took about 1m 40s: $ time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044 real 1m41,644s user 0m0,926s sys 0m0,087s * same test with all those patches reverted - and it took 8 seconds: $ sudo systemctl restart devstack@q-svc; sleep 5; time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044 real 0m8,131s user 0m0,907s sys 0m0,120s After a bit of digging I found out that most of the slowdown came from calls of the method self._handle_deprecated_rule(default) in [3]. When I commented that one line I had results same like without secure-rbac patches: $ sudo systemctl restart devstack@q-svc; sleep 5; time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044 real 0m7,875s user 0m0,931s sys 0m0,087s @Lance, can You maybe take a look at it and help me understand if we are doing something wrong with those new rules in Neutron? Or maybe there is a bug in oslo_policy? [1] https://review.opendev.org/q/topic:secure-rbac+project:openstack/neutron [2] https://bugs.launchpad.net/neutron/+bug/1913718 [3] https://github.com/openstack/oslo.policy/blob/e103baa002e54303b08630c436dfc7... -- Slawek Kaplonski Principal Software Engineer Red Hat
I'm able to recreate this locally. I updated the bug, but I didn't triage it. I left a comment with some thoughts on where the extra processing is taking place, but I need to parse more of neutron's policy enforcement engine. i don't think nova is doing per-attribute policy enforcement, which is where I think the extra cycles are in neutron. Thanks for raising the issue, Slawek. On Fri, Jan 29, 2021 at 10:14 AM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
In Neutron we are merging more and more patches related to the secure rbac policies [1]. Recently we noticed that our neutron-rally-task job is failing very often [2].
I did some tests and here is what were my results:
* on latest neutron, with about 12 patches related to secure rbac merged - list of 4k ports took about 1m 40s:
$ time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044
real 1m41,644s user 0m0,926s sys 0m0,087s
* same test with all those patches reverted - and it took 8 seconds:
$ sudo systemctl restart devstack@q-svc; sleep 5; time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044
real 0m8,131s user 0m0,907s sys 0m0,120s
After a bit of digging I found out that most of the slowdown came from calls of the method self._handle_deprecated_rule(default) in [3]. When I commented that one line I had results same like without secure-rbac patches:
$ sudo systemctl restart devstack@q-svc; sleep 5; time neutron port-list | wc -l neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. 4044
real 0m7,875s user 0m0,931s sys 0m0,087s
@Lance, can You maybe take a look at it and help me understand if we are doing something wrong with those new rules in Neutron? Or maybe there is a bug in oslo_policy?
[1] https://review.opendev.org/q/topic:secure-rbac+project:openstack/neutron [2] https://bugs.launchpad.net/neutron/+bug/1913718 [3] https://github.com/openstack/oslo.policy/blob/e103baa002e54303b08630c436dfc7...
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (2)
-
Lance Bragstad
-
Slawek Kaplonski