[openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

Morgan Fainberg morgan.fainberg at gmail.com
Tue Apr 12 15:19:35 UTC 2016


Sorry Missed the copy/paste of links:

https://bugs.launchpad.net/keystone/+bug/1567403 [0]
https://bugs.launchpad.net/keystone/+bug/1567413 [1]

[0]
https://review.openstack.org/#/q/I4857cfe1e62d54c3c89a0206ffc895c4cf681ce5,n,z
[1] https://review.openstack.org/#/c/304688/

--Morgan

On Tue, Apr 12, 2016 at 8:16 AM, Morgan Fainberg <morgan.fainberg at gmail.com>
wrote:

> Fixes have been proposed for both of these bugs.
>
> Cheers,
> --Morgan
>
> On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova <dbelova at mirantis.com>
> wrote:
>
>> Matt,
>>
>> Thanks for sharing the information about your benchmark. Indeed we need
>> to follow up on this topic (I'll attend the summit). Let's try to collect
>> as much information as possible prior Austin to have more facts to operate.
>> I'll try to figure out why local context cache did not work at least on my
>> environment, and, due to the results, most probably it did not act as
>> supposed during your benchmarking as well.
>>
>> Cheers,
>> Dina
>>
>> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer <matt at mattfischer.com>
>> wrote:
>>
>>> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova <dbelova at mirantis.com>
>>> wrote:
>>>
>>>> Hey, openstackers!
>>>>
>>>> Recently I was trying to profile Keystone (OpenStack Liberty vs Mitaka)
>>>> using this set of changes
>>>> <https://review.openstack.org/#/q/topic:osprofiler-support-in-keystone+status:open> (that's
>>>> currently on review - some final steps are required there to finish the
>>>> work) and OSprofiler.
>>>>
>>>> Some preliminary results (all in one OpenStack node) can be found here
>>>> <http://docs.openstack.org/developer/performance-docs/test_results/keystone/all-in-one/index.html> (raw
>>>> OSprofiler reports are not yet merged to some place and can be found
>>>> here <https://review.openstack.org/#/c/299514/>). The full plan
>>>> <http://docs.openstack.org/developer/performance-docs/test_plans/keystone/plan.html#keystone-performance> of
>>>> what's going to be tested  can be found in the docs as well. In short I
>>>> wanted to take a look how does Keystone changed its DB/Cache usage from
>>>> Liberty to Mitaka, keeping in mind that there were several changes
>>>> introduced:
>>>>
>>>>    - federation support was added (and made DB scheme a bit more
>>>>    complex)
>>>>    - Keystone moved to oslo.cache usage
>>>>    - local context cache was introduced during Mitaka
>>>>
>>>> First of all - *good job on making Keystone less DB-extensive in case
>>>> of cache turned on*! If Keystone caching is turned on, number of DB
>>>> queries done to Keystone DB in Mitaka is averagely twice less than in
>>>> Liberty, comparing the same requests and topologies. Thanks Keystone
>>>> community to make it happen :)
>>>>
>>>> Although, I faced *two strange issues* during my experiments, and I'm
>>>> kindly asking you, folks, to help me here:
>>>>
>>>>    - I've created #1567403
>>>>    <https://bugs.launchpad.net/keystone/+bug/1567403> bug to share
>>>>    information - when I turned caching on, local context cache should cache
>>>>    identical per API requests function calls not to ping Memcache too often.
>>>>    Although I faced such calls, Keystone still used Memcache to gather this
>>>>    information. May someone take a look on this and help me figure out what am
>>>>    I observing? At the first sight local context cache should work ok, but for
>>>>    some reason I do not see it's being used.
>>>>    - One more filed bug - #1567413
>>>>    <https://bugs.launchpad.net/keystone/+bug/1567413> - is about a bit
>>>>    opposite situation :) When I turned cache off explicitly in the
>>>>    keystone.conf file, I still observed some of the values being fetched from
>>>>    Memcache... Your help is very appreciated!
>>>>
>>>> Thanks in advance and sorry for a long email :)
>>>>
>>>> Cheers,
>>>> Dina
>>>>
>>>>
>>> Dina,
>>>
>>> Thanks for starting this conversation. I had some weird perf results
>>> comparing L to an RC release of Mitaka, but I was holding them until
>>> someone else confirmed what I saw. I'm testing token creation and
>>> validation. From what I saw, token validation slowed down in Mitaka. After
>>> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
>>> it was in Liberty. That implies more caching but 8x is a lot and even
>>> memcache references are not free.
>>>
>>> I know some of the Keystone folks are looking into this so it will be
>>> good to follow-up on it. Maybe we could talk about this at the summit?
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Dina Belova
>>
>> Software Engineer
>>
>> Mirantis Inc.
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160412/cccd09f1/attachment.html>


More information about the OpenStack-dev mailing list