Open Stack

On Tue, Jul 16, 2013 at 4:01 AM, Kieran Spear <kispear at gmail.com> wrote:
>
> On 16/07/2013, at 1:10 AM, Adam Young <ayoung at redhat.com> wrote:
>> On 07/15/2013 04:06 AM, Kieran Spear wrote:
>>> Hi all,
>>>
>>> I want to backport the fix for the "Token List in Memcache can consume
>>> an entire memcache page" bug[1] to Grizzly, but I had a couple of
>>> questions:
>>>
>>> 1. Why do we need to store the entire token data in the
>>> usertoken-<userid> key? This data always seems to be hashed before
>>> indexing into the 'token-<tokenid>' keys anyway. The size of the
>>> memcache data for a user's token list currently grows by 4k every time
>>> a new PKI token is created. It doesn't take long to hit 1MB at this
>>> rate even with the above fix.
>> Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit.
>
> In this case it looks like we're taking a hit from both, since the PKI token "id" from the user token index is retrieved, then hashed and then that key is used to retrieve the token from the "tokens-%s" page anyway.
>
>>
>>
>>>
>>> 2. Every time it creates a new token, Keystone loads each token from
>>> the user's token list with a separate memcache call so it can throw it
>>> away if it's expired. This seems excessive. Is it anything to worry
>>> about? If it just checked the first two tokens you'd get the same
>>> effect on a longer time scale.
>>>
>>> I guess part of the answer is to decrease our token expiry time, which
>>> should mitigate both issues. Failing that we'd consider moving to the
>>> SQL backend.
>> HOw about doing both?  But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well.  No silver bullet, I am afraid.
>
> I think we're going to stick with memcache for now (the devil we know :)). With (1) and (2) fixed and the token expiration time tweaked I think memcache will do okay.
>
> Kieran
>
>>
>>>
>>> Cheers,
>>> Kieran
>>>
>>> [1] https://bugs.launchpad.net/keystone/+bug/1171985
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi Kieran,

I've looked into the potential bug you described and it appears that
there has been a change in the master branch to support the idea of
pluggable token providers (much better implementation than the driver
being responsible for the token itself).  This change modified how the
memcache driver stored the IDs, and performed the CMS hashing function
when the manager returned the token_id to the driver, instead of
in-line within the driver.  The original fix should have been correct
in hashing the PKI token to the short-form "ID".  Your fix to simply
hash the tokens is the correct one and more closely mirrors how the
original fix was implemented.

If you are interested in the reviews that implement the new pluggable
provider(s): https://review.openstack.org/#/c/33858/ (V3) and
https://review.openstack.org/#/c/34421/ (V2.0).

Going with the shorter TTL on the Tokens is a good idea for various
reasons depending on the token driver.  I know that the SQL driver
(provided you cleanup expired tokens) has worked well for my company,
but I want to move to the memcache driver soon.

Cheers,
Morgan Fainberg