<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Sounds good! Just remember that prior to the fix I posted there, for each token in the user’s index, it incurred a round-trip to memcached to validate the token wasn’t expired. This change makes it so that there are significantly less trips from keystone to memcached.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">If this doesn’t 100% solve the issue, we should start digging further into what is going on, but I am confident this will (at the very least) help a reasonable amount.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">—Morgan </div> <br><p style="color:#A0A0A8;">On January 11, 2014 at 19:04:59, Jonathan Proulx (<a href="mailto://jon@jonproulx.com">jon@jonproulx.com</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div> On Sat, Jan 11, 2014 at 8:24 PM, Morgan Fainberg <m@metacloud.com> wrote:
<br>> Hi Jon,
<br>>
<br>> I have published a patch set that I hope will help to address this issue:
<br>> https://review.openstack.org/#/c/66149/ . If you need this in another
<br>> format, please let me know.
<br>
<br>That's a fine format, also I love patches that only touch one file (plus tests).
<br>
<br>> The only caveat is that you should expect the maximum number of tokens
<br>> per-user to drop by as much as 50% due to storing extra data about the token
<br>> (avoiding the need to ask memcache about each token on issuance of a new
<br>> one). I still recommend keeping the expires time down to ~3600 seconds or
<br>> so. I also recommend doing a flush (like you did before) when deploying
<br>> this change.
<br>>
<br>> Let me know if this helps (or if you have any issues with it). Feel free to
<br>> respond via email or comment on the review. Disclaimer: I have not
<br>> performed functional performance tests on this code, just some initial
<br>> cleanup and change of logic that should help minimize external calls.
<br>
<br>I'll try it out tomorrow and let you know. The memcache item counts at
<br>failure are 1/32nd to 1/16th what you were suggesting as an upper
<br>bound, so I'm not sure this is quite the issue I'm having, but I hope
<br>it is
<br>
<br>By juggling a variety of things I've managed to stablize things at a
<br>poor but functional stage . I have keystone running behind apache with
<br>12 processes set though well fewer than that seem to actually run at
<br>any given time. I needed to bring the ttl down to 600s, which makes
<br>the dashboard hard to use :) It's difficult to draw conclusions from
<br>this comparison, but right after flushing and restarting 'nova list'
<br>on a small tenant takes about 1.5sec to return, at the stable state
<br>(about 1 ttl later) this same operation takes on the order of 30sec
<br>(25-40). At this point memcache only reports just over 1k items,
<br>which doesn't seem like a lot to me (at 3600s ttl it gets upto about
<br>2.5k but things stop working so I'd guess it locks up around 1800s but
<br>I've not full explored the range. .
<br>
<br>Thanks,
<br>-Jon
<br></div></div></span></blockquote></body></html>