[openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

Morgan Fainberg morgan.fainberg at gmail.com
Tue Apr 12 15:16:56 UTC 2016


Fixes have been proposed for both of these bugs.

Cheers,
--Morgan

On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova <dbelova at mirantis.com> wrote:

> Matt,
>
> Thanks for sharing the information about your benchmark. Indeed we need to
> follow up on this topic (I'll attend the summit). Let's try to collect as
> much information as possible prior Austin to have more facts to operate.
> I'll try to figure out why local context cache did not work at least on my
> environment, and, due to the results, most probably it did not act as
> supposed during your benchmarking as well.
>
> Cheers,
> Dina
>
> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer <matt at mattfischer.com>
> wrote:
>
>> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova <dbelova at mirantis.com>
>> wrote:
>>
>>> Hey, openstackers!
>>>
>>> Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
>>> using this set of changes
>>> <https://review.openstack.org/#/q/topic:osprofiler-support-in-keystone+status:open> (that's
>>> currently on review - some final steps are required there to finish the
>>> work) and OSprofiler.
>>>
>>> Some preliminary results (all in one OpenStack node) can be found here
>>> <http://docs.openstack.org/developer/performance-docs/test_results/keystone/all-in-one/index.html> (raw
>>> OSprofiler reports are not yet merged to some place and can be found
>>> here <https://review.openstack.org/#/c/299514/>). The full plan
>>> <http://docs.openstack.org/developer/performance-docs/test_plans/keystone/plan.html#keystone-performance> of
>>> what's going to be tested  can be found in the docs as well. In short I
>>> wanted to take a look how does Keystone changed its DB/Cache usage from
>>> Liberty to Mitaka, keeping in mind that there were several changes
>>> introduced:
>>>
>>>    - federation support was added (and made DB scheme a bit more
>>>    complex)
>>>    - Keystone moved to oslo.cache usage
>>>    - local context cache was introduced during Mitaka
>>>
>>> First of all - *good job on making Keystone less DB-extensive in case
>>> of cache turned on*! If Keystone caching is turned on, number of DB
>>> queries done to Keystone DB in Mitaka is averagely twice less than in
>>> Liberty, comparing the same requests and topologies. Thanks Keystone
>>> community to make it happen :)
>>>
>>> Although, I faced *two strange issues* during my experiments, and I'm
>>> kindly asking you, folks, to help me here:
>>>
>>>    - I've created #1567403
>>>    <https://bugs.launchpad.net/keystone/+bug/1567403> bug to share
>>>    information - when I turned caching on, local context cache should cache
>>>    identical per API requests function calls not to ping Memcache too often.
>>>    Although I faced such calls, Keystone still used Memcache to gather this
>>>    information. May someone take a look on this and help me figure out what am
>>>    I observing? At the first sight local context cache should work ok, but for
>>>    some reason I do not see it's being used.
>>>    - One more filed bug - #1567413
>>>    <https://bugs.launchpad.net/keystone/+bug/1567413> - is about a bit
>>>    opposite situation :) When I turned cache off explicitly in the
>>>    keystone.conf file, I still observed some of the values being fetched from
>>>    Memcache... Your help is very appreciated!
>>>
>>> Thanks in advance and sorry for a long email :)
>>>
>>> Cheers,
>>> Dina
>>>
>>>
>> Dina,
>>
>> Thanks for starting this conversation. I had some weird perf results
>> comparing L to an RC release of Mitaka, but I was holding them until
>> someone else confirmed what I saw. I'm testing token creation and
>> validation. From what I saw, token validation slowed down in Mitaka. After
>> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
>> it was in Liberty. That implies more caching but 8x is a lot and even
>> memcache references are not free.
>>
>> I know some of the Keystone folks are looking into this so it will be
>> good to follow-up on it. Maybe we could talk about this at the summit?
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
>
> Best regards,
>
> Dina Belova
>
> Software Engineer
>
> Mirantis Inc.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160412/713decc8/attachment.html>


More information about the OpenStack-dev mailing list