Open Stack

On 17/07/2013, at 12:26 AM, Morgan Fainberg <m at metacloud.com> wrote:

> On Tue, Jul 16, 2013 at 4:01 AM, Kieran Spear <kispear at gmail.com> wrote:
>> 
>> On 16/07/2013, at 1:10 AM, Adam Young <ayoung at redhat.com> wrote:
>>> On 07/15/2013 04:06 AM, Kieran Spear wrote:
>>>> Hi all,
>>>> 
>>>> I want to backport the fix for the "Token List in Memcache can consume
>>>> an entire memcache page" bug[1] to Grizzly, but I had a couple of
>>>> questions:
>>>> 
>>>> 1. Why do we need to store the entire token data in the
>>>> usertoken-<userid> key? This data always seems to be hashed before
>>>> indexing into the 'token-<tokenid>' keys anyway. The size of the
>>>> memcache data for a user's token list currently grows by 4k every time
>>>> a new PKI token is created. It doesn't take long to hit 1MB at this
>>>> rate even with the above fix.
>>> Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit.
>> 
>> In this case it looks like we're taking a hit from both, since the PKI token "id" from the user token index is retrieved, then hashed and then that key is used to retrieve the token from the "tokens-%s" page anyway.
>> 
>>> 
>>> 
>>>> 
>>>> 2. Every time it creates a new token, Keystone loads each token from
>>>> the user's token list with a separate memcache call so it can throw it
>>>> away if it's expired. This seems excessive. Is it anything to worry
>>>> about? If it just checked the first two tokens you'd get the same
>>>> effect on a longer time scale.
>>>> 
>>>> I guess part of the answer is to decrease our token expiry time, which
>>>> should mitigate both issues. Failing that we'd consider moving to the
>>>> SQL backend.
>>> HOw about doing both?  But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well.  No silver bullet, I am afraid.
>> 
>> I think we're going to stick with memcache for now (the devil we know :)). With (1) and (2) fixed and the token expiration time tweaked I think memcache will do okay.
>> 
>> Kieran
>> 
>>> 
>>>> 
>>>> Cheers,
>>>> Kieran
>>>> 
>>>> [1] https://bugs.launchpad.net/keystone/+bug/1171985
>>>> 
> 
> Hi Kieran,
> 
> I've looked into the potential bug you described and it appears that
> there has been a change in the master branch to support the idea of
> pluggable token providers (much better implementation than the driver
> being responsible for the token itself).  This change modified how the
> memcache driver stored the IDs, and performed the CMS hashing function
> when the manager returned the token_id to the driver, instead of
> in-line within the driver.  The original fix should have been correct
> in hashing the PKI token to the short-form "ID".  Your fix to simply
> hash the tokens is the correct one and more closely mirrors how the
> original fix was implemented.
> 
> If you are interested in the reviews that implement the new pluggable
> provider(s): https://review.openstack.org/#/c/33858/ (V3) and
> https://review.openstack.org/#/c/34421/ (V2.0).
> 
> Going with the shorter TTL on the Tokens is a good idea for various
> reasons depending on the token driver.  I know that the SQL driver
> (provided you cleanup expired tokens) has worked well for my company,
> but I want to move to the memcache driver soon.

Thanks for the info. Good to know this is fixed on master. Spotted one possible upgrade issue in the V3 patch which I've submitted a bug for:

https://bugs.launchpad.net/keystone/+bug/1202050

And a bug for getting my one-line fix into stable/grizzly:

https://bugs.launchpad.net/keystone/+bug/1202053

Cheers,
Kieran

> 
> Cheers,
> Morgan Fainberg