[openstack-dev] [keystone][performance][profiling] Profiling Mitaka Keystone: some results and asking for a help

Dina Belova dbelova at mirantis.com
Tue Apr 12 15:51:12 UTC 2016


Thank you Morgan for quick fixes proposed!

On Tue, Apr 12, 2016 at 6:19 PM, Morgan Fainberg <morgan.fainberg at gmail.com>
wrote:

> Sorry Missed the copy/paste of links:
>
> https://bugs.launchpad.net/keystone/+bug/1567403 [0]
> https://bugs.launchpad.net/keystone/+bug/1567413 [1]
>
> [0]
> https://review.openstack.org/#/q/I4857cfe1e62d54c3c89a0206ffc895c4cf681ce5,n,z
> [1] https://review.openstack.org/#/c/304688/
>
> --Morgan
>
> On Tue, Apr 12, 2016 at 8:16 AM, Morgan Fainberg <
> morgan.fainberg at gmail.com> wrote:
>
>> Fixes have been proposed for both of these bugs.
>>
>> Cheers,
>> --Morgan
>>
>> On Tue, Apr 12, 2016 at 12:38 AM, Dina Belova <dbelova at mirantis.com>
>> wrote:
>>
>>> Matt,
>>>
>>> Thanks for sharing the information about your benchmark. Indeed we need
>>> to follow up on this topic (I'll attend the summit). Let's try to collect
>>> as much information as possible prior Austin to have more facts to operate.
>>> I'll try to figure out why local context cache did not work at least on my
>>> environment, and, due to the results, most probably it did not act as
>>> supposed during your benchmarking as well.
>>>
>>> Cheers,
>>> Dina
>>>
>>> On Mon, Apr 11, 2016 at 10:57 PM, Matt Fischer <matt at mattfischer.com>
>>> wrote:
>>>
>>>> On Mon, Apr 11, 2016 at 8:11 AM, Dina Belova <dbelova at mirantis.com>
>>>> wrote:
>>>>
>>>>> Hey, openstackers!
>>>>>
>>>>> Recently I was trying to profile Keystone (OpenStack Liberty vs
>>>>> Mitaka) using this set of changes
>>>>> <https://review.openstack.org/#/q/topic:osprofiler-support-in-keystone+status:open> (that's
>>>>> currently on review - some final steps are required there to finish the
>>>>> work) and OSprofiler.
>>>>>
>>>>> Some preliminary results (all in one OpenStack node) can be found here
>>>>> <http://docs.openstack.org/developer/performance-docs/test_results/keystone/all-in-one/index.html> (raw
>>>>> OSprofiler reports are not yet merged to some place and can be found
>>>>> here <https://review.openstack.org/#/c/299514/>). The full plan
>>>>> <http://docs.openstack.org/developer/performance-docs/test_plans/keystone/plan.html#keystone-performance> of
>>>>> what's going to be tested  can be found in the docs as well. In short I
>>>>> wanted to take a look how does Keystone changed its DB/Cache usage from
>>>>> Liberty to Mitaka, keeping in mind that there were several changes
>>>>> introduced:
>>>>>
>>>>>    - federation support was added (and made DB scheme a bit more
>>>>>    complex)
>>>>>    - Keystone moved to oslo.cache usage
>>>>>    - local context cache was introduced during Mitaka
>>>>>
>>>>> First of all - *good job on making Keystone less DB-extensive in case
>>>>> of cache turned on*! If Keystone caching is turned on, number of DB
>>>>> queries done to Keystone DB in Mitaka is averagely twice less than in
>>>>> Liberty, comparing the same requests and topologies. Thanks Keystone
>>>>> community to make it happen :)
>>>>>
>>>>> Although, I faced *two strange issues* during my experiments, and I'm
>>>>> kindly asking you, folks, to help me here:
>>>>>
>>>>>    - I've created #1567403
>>>>>    <https://bugs.launchpad.net/keystone/+bug/1567403> bug to share
>>>>>    information - when I turned caching on, local context cache should cache
>>>>>    identical per API requests function calls not to ping Memcache too often.
>>>>>    Although I faced such calls, Keystone still used Memcache to gather this
>>>>>    information. May someone take a look on this and help me figure out what am
>>>>>    I observing? At the first sight local context cache should work ok, but for
>>>>>    some reason I do not see it's being used.
>>>>>    - One more filed bug - #1567413
>>>>>    <https://bugs.launchpad.net/keystone/+bug/1567413> - is about a
>>>>>    bit opposite situation :) When I turned cache off explicitly in the
>>>>>    keystone.conf file, I still observed some of the values being fetched from
>>>>>    Memcache... Your help is very appreciated!
>>>>>
>>>>> Thanks in advance and sorry for a long email :)
>>>>>
>>>>> Cheers,
>>>>> Dina
>>>>>
>>>>>
>>>> Dina,
>>>>
>>>> Thanks for starting this conversation. I had some weird perf results
>>>> comparing L to an RC release of Mitaka, but I was holding them until
>>>> someone else confirmed what I saw. I'm testing token creation and
>>>> validation. From what I saw, token validation slowed down in Mitaka. After
>>>> doing my benchmark runs, the traffic to memcache was 8x in Mitaka from what
>>>> it was in Liberty. That implies more caching but 8x is a lot and even
>>>> memcache references are not free.
>>>>
>>>> I know some of the Keystone folks are looking into this so it will be
>>>> good to follow-up on it. Maybe we could talk about this at the summit?
>>>>
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>>
>>> Dina Belova
>>>
>>> Software Engineer
>>>
>>> Mirantis Inc.
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Dina Belova

Software Engineer

Mirantis Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160412/7559df6a/attachment.html>


More information about the OpenStack-dev mailing list