Open Stack

On Mon, Jul 15, 2013 at 1:06 AM, Kieran Spear <kispear at gmail.com> wrote:

> Hi all,
>
> I want to backport the fix for the "Token List in Memcache can consume
> an entire memcache page" bug[1] to Grizzly, but I had a couple of
> questions:
>
> 1. Why do we need to store the entire token data in the
> usertoken-<userid> key? This data always seems to be hashed before
> indexing into the 'token-<tokenid>' keys anyway. The size of the
> memcache data for a user's token list currently grows by 4k every time
> a new PKI token is created. It doesn't take long to hit 1MB at this
> rate even with the above fix.
>
> 2. Every time it creates a new token, Keystone loads each token from
> the user's token list with a separate memcache call so it can throw it
> away if it's expired. This seems excessive. Is it anything to worry
> about? If it just checked the first two tokens you'd get the same
> effect on a longer time scale.
>
> I guess part of the answer is to decrease our token expiry time, which
> should mitigate both issues. Failing that we'd consider moving to the
> SQL backend.
>
> Cheers,
> Kieran
>
> [1] https://bugs.launchpad.net/keystone/+bug/1171985
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Hi Kieran,

I'd be happy to help you with the backporting of this fix if you need any
assistance.

Here are some quick answers to your questions:

1. The individual tokens are stored in independent keys in their entirety,
since the memcache backend replaces the SQL backend.  In the
usertoken-<userid> we should only be storing the hashed value.  It looks
like there is a bug there (I'll check and get a fix proposed in gerrit
today to address this).

2.  There has been a lot of discussion on this topic and some other
solutions have been proposed (such as use a clock-mechanism to expire
tokens, etc).  However, to stay compatible with the current business logic,
which requires the ability to revoke subsets of tokens for the user
(instead. all tokens for the user on every change to the user's
roles/project associations, etc), a list of current tokens for the user
must be maintained.  Ideally, the token list shouldn't ever be massive,
meaning the extra calls to memcache should be limited.  The general plan is
to develop another approach to solve this issue without the need of keeping
a user-token-list, but it just wasn't viewed as possible for Havana.
 Checking the first two tokens wouldn't exactly solve the issue, as tokens
can in theory be requested with varying expiration times; this means that
tokens in the middle of the list could be expired whereas the tokens at the
start of the list are invalid.

Keeping the expiry time low is definitely a good idea.


Let me know if you want any further details.  I'd be happy to elaborate /
fill-in more.

Cheers,
Morgan Fainberg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130715/2a079c28/attachment.html>