<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 1, 2013 at 4:59 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 03/01/2013 01:18 PM, Vishvananda Ishaya wrote:<br>
> Hi Everyone,<br>
><br>
> So I've been doing some profiling of api calls against devstack and I've discovered that a significant portion of time spent is in the auth_token middleware validating the PKI token. There is code to turn on caching of the token if memcache is enabled, but this seems like overkill in most cases. We should be caching the token in memory by default. Fortunately, nova has some nifty code that will use an in-memory cache if memcached isn't available.<br>
<br>
</div>We gave up on PKI in Folsom after weeks of trouble with it:<br>
<br>
* Unstable -- Endpoints would stay up >24 hours but after around 24<br>
hours (sometimes sooner), the endpoint would stop working properly with<br>
the server user suddenly returned a 401 when trying to authenticate a<br>
token. Restarting the endpoint with a service nova-api restart gets rid<br>
of the 401 Unauthorized for a few hours and then it happens again.<br></blockquote><div><br></div><div style>Obviously that's not acceptable behavior; is there a bug tracking this issue? I poked around but didn't see anything related to unexpected 401's.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
* Unable to use memcache with PKI. The token was longer than the maximum<br>
memcache key and resulted in errors on every request. The solution for<br>
this was to hash the CMS token and use hash as a key in memcache, but<br>
unfortunately this solution wasn't backported to Folsom Keystone --<br>
partly I think because the auth_token middleware was split out into the<br>
keystoneclient during Grizzly.<br></blockquote><div><br></div><div style>I believe keystoneclient.middleware.auth_token supports this and is backwards compatible with essex and folsom-- any reason why you couldn't utilize the latest client / middleware?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
In any case, the above two things make PKI unusable in Folsom.<br>
<br>
We fell back on UUID tokens -- the default in Folsom. Unfortunately,<br>
there are serious performance issues with this approach as well. Every<br>
single request to an endpoint results in multiple requests to Keystone,<br>
which bogs down the system.<br></blockquote><div><br></div><div style>+1; the best solution to this is to have clients cache tokens until they're expired. keystoneclient does this if keyring support is enabled.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
In addition to the obvious roundtrip issues, with just 26 users in a<br>
test cloud, in 3 weeks there are over 300K records in the tokens table<br>
on a VERY lightly used cloud. Not good. Luckily, we use multi-master<br>
MySQL replication (Galera) with excellent write rates spread across four<br>
cluster nodes, but this scale of writes for such a small test cluster is<br>
worrying to say the least.<br></blockquote><div><br></div><div style>At the moment, you're free to do whatever you want with expired tokens and you won't otherwise impact the deployment. Delete them, archive them, whatever. Joe Breu suggested adding a keystone-manage command to copy expired tokens into a tokens_archive table, that could be called periodically to clean up... I'm certainly in favor of that. Perhaps something like:</div>
<div style><br></div><div style> $ keystone-manage flush-expired-tokens [--strategy=archive|delete]</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Although not related to PKI, I've also noticed that due to the decision<br>
to use a denormalized schema in the users table with the "extra" column<br>
storing a JSON-encoded blob of data including the user's default tenant<br>
and enabled flag is a horrible performance problem. Hope that v3<br>
Keystone has corrected these issues in the SQL driver.<br></blockquote><div><br></div><div style>We've made several such improvements. We still use the 'extra' columns, but hopefully not for anything that should be indexed.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
><br>
> 1) Shim the code into the wsgi stack using the configuration options designed for swift:<br>
><br>
> <a href="https://review.openstack.org/23236" target="_blank">https://review.openstack.org/23236</a><br>
><br>
> This is my least favorite option since changing paste config is a pain for deployers and it doesn't help any of the other projects.<br>
<br>
</div>Meh, whether you add options to a config file or a paste INI file it's<br>
the same pain for deployers :) But generally agree with you.<br>
<div class="im"><br>
> 2) Copy the code into keystoneclient:<br>
><br>
> <a href="https://review.openstack.org/23307" target="_blank">https://review.openstack.org/23307</a><br>
><br>
> 3) Move memorycache into oslo and sync it to nova and keystoneclient:<br>
><br>
> <a href="https://review.openstack.org/23306" target="_blank">https://review.openstack.org/23306</a><br>
> <a href="https://review.openstack.org/23308" target="_blank">https://review.openstack.org/23308</a><br>
> <a href="https://review.openstack.org/23309" target="_blank">https://review.openstack.org/23309</a><br>
><br>
> I think 3) is the right long term move, but I'm not sure if this appropriate considering how close we are to the grizzly release, so if we want to do 2) immediately and postpone 3) until H, that is fine with me.<br>
<br>
</div>Well, I think 3) is the right thing to do in any case, and can be done<br>
in oslo regardless of Nova's RC status.<br></blockquote><div><br></div><div style>Since the feature has merged to oslo, I take it that oslo isn't abiding by feature freeze? I'm happy to see it utilized in keystoneclient.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Not sure that 2) is really all that useful. If you are in any serious<br>
production environment, you're going to be using memcached anyway.<br>
<br>
Best,<br>
-jay<br>
<div class="HOEnZb"><div class="h5"><br>
> Thoughts?<br>
><br>
> Vish<br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>