[qa][openstackclient] Debugging devstack slowness

Alex Schultz aschultz at redhat.com
Mon Jul 29 13:40:11 UTC 2019


On Fri, Jul 26, 2019 at 5:57 PM Clark Boylan <cboylan at sapwetik.org> wrote:

> Today I have been digging into devstack runtime costs to help Donny Davis
> understand why tempest jobs sometimes timeout on the FortNebula cloud. One
> thing I discovered was that the keystone user, group, project, role, and
> domain setup [0] can take many minutes [1][2] (in the examples here almost
> 5).
>
> I've rewritten create_keystone_accounts to be a python tool [3] and get
> the runtime for that subset of setup from ~100s to ~9s [4].  I imagine that
> if we applied this to the other create_X_accounts functions we would see
> similar results.
>
> I think this is so much faster because we avoid repeated costs in
> openstack client including: python process startup, pkg_resource disk
> scanning to find entrypoints, and needing to convert names to IDs via the
> API every time osc is run. Given my change shows this can be so much
> quicker is there any interest in modifying devstack to be faster here? And
> if so what do we think an appropriate approach would be?
>
>
In tripleo, we've also run into the same thing for other actions. While you
can do bulk openstack client actions[0], it's not the best thing if you
need to create a resource and fetch an ID for a subsequent action. We
ported our post-installation items to python[1] and noticed a dramatic
improvement as well.  It might be beneficial to maybe add some caching into
openstackclient so that the startup cost isn't so large every time?

[0] https://review.opendev.org/#/c/521146/
[1] https://review.opendev.org/#/c/614540/


> [0]
> https://opendev.org/openstack/devstack/src/commit/6aeaceb0c4ef078d028fb6605cac2a37444097d8/stack.sh#L1146-L1161
> [1]
> http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_31_04_488228
> [2]
> http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output.txt.gz#_2019-07-26_12_35_53_445059
> [3] https://review.opendev.org/#/c/673108/
> [4]
> http://logs.openstack.org/08/673108/6/check/devstack-xenial/a4107d0/job-output.txt.gz#_2019-07-26_23_18_37_211013
>
> Note the jobs compared above all ran on rax-dfw.
>
> Clark
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190729/88347645/attachment.html>


More information about the openstack-discuss mailing list