Open Stack

 On Sat, Jan 11, 2014 at 8:24 PM, Morgan Fainberg <m at metacloud.com> wrote:
> Hi Jon,
>
> I have published a patch set that I hope will help to address this issue:
> https://review.openstack.org/#/c/66149/ . If you need this in another
> format, please let me know.

That's a fine format, also I love patches that only touch one file (plus tests).

> The only caveat is that you should expect the maximum number of tokens
> per-user to drop by as much as 50% due to storing extra data about the token
> (avoiding the need to ask memcache about each token on issuance of a new
> one).  I still recommend keeping the expires time down to ~3600 seconds or
> so.  I also recommend doing a flush (like you did before) when deploying
> this change.
>
> Let me know if this helps (or if you have any issues with it).  Feel free to
> respond via email or comment on the review.  Disclaimer: I have not
> performed functional performance tests on this code, just some initial
> cleanup and change of logic that should help minimize external calls.

I'll try it out tomorrow and let you know. The memcache item counts at
failure are 1/32nd to 1/16th what you were suggesting as an upper
bound, so I'm not sure this is quite the issue I'm having, but I hope
it is

By juggling a variety of things I've managed to stablize things at a
poor but functional stage . I have keystone running behind apache with
12 processes set though well fewer than that seem to actually run at
any given time. I needed to bring the ttl down to 600s, which makes
the dashboard hard to use :)  It's difficult to draw conclusions from
this comparison, but right after flushing and restarting 'nova list'
on a small tenant takes about 1.5sec to return, at the stable state
(about 1 ttl later) this same operation takes on the order of 30sec
(25-40).  At this point memcache only reports just over 1k items,
which doesn't seem like a lot to me (at 3600s ttl it gets upto about
2.5k but things stop working so I'd guess it locks up around 1800s but
I've not full explored the range.  .

Thanks,
-Jon