[Keystone] Performance degradation

Morgan Fainberg morgan.fainberg at gmail.com
Sun Jun 16 15:43:55 UTC 2019


Hi Gary,

A couple things come to mind right off the bat that I want to ensure are
addressed:

1) Is caching enabled and is the memcache server in-fact accessible from
the keystone process? Keystone is developed assuming you have caching
enabled (we will be examining, during Train, making this a requirement
instead of "strongly encouraged").

2) Did you swap from uuid to Fernet between the deployments? Unfortunately,
fernet is slower than UUID. We opted for fernet and taking the performance
hit in light of the significantly better token management/maintenance for
long-term running clouds. UUID token provider was removed as of rocky.

3) I am curious about the scenario you have built for rally: Is this
scenario a real-world-ish scenario that you are doing on a regular basis? I
want to be clear we are troubleshooting real-world(ish) scenarios and not
synthetic problems that only occur in test scenarios; there are options to
streamline test scenarios separately from real-world use-cases. Tell me
more about the rally scenario. I also want to point out that I've been
unable to get any real information from the attached files (they seem to be
broken).

4) Is this running under mod_wsgi? uwsgi? is there a lot of other process
space contention if under mod_wsgi in apache? (There isn't a lot of
information about your configuration in the posed question), Configuration
information, deployment in the container, etc helps us understand what is
going on.

Thanks!

On Sun, Jun 16, 2019 at 1:48 AM Gary Kotton <gkotton at vmware.com> wrote:

> Hi,
>
> Over the last few weeks we have being doing performance tests on the Stein
> version and have seen a notable performance degradation. Further
> investigation showed that Keystone seems to be the bottleneck.
>
> An example of this is running a Rally test that creates a keystone tenant
> with users. We see that in Queens this takes 20 seconds to complete the
> whole iteration whilst Stein takes 30 seconds.
>
> We have done the following tests:
>
>    1. Vanilla devstack Queens vs Steins
>    2. In our Stein version e have swapped out the Keystone Stein
>    container with the Keystone Queens container and the numbers are
>    considerably better too (please note the same keystone configuration is
>    used with regards to processes/threads etc.)
>
> Are any folks familiar with what may cause this?
>
> Are there any performance improvement suggestions or hints?
>
> Thanks
>
> Gary
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190616/3564103f/attachment.html>


More information about the openstack-discuss mailing list