<div dir="ltr">Actually, Bluehost is using it in production. We couldn't get past a couple thousand nodes without it because of the amount of requests that the quantum network driver produces (5 every periodic interval per compute node). It does have some problems if one tenant builds up a large list of tokens, but other than that it has been great for us. I think our deployment is somewhere around 15,000 nodes right now and it is still holding up strong. It is MUCH more performant than just a plain SQL backend.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 28, 2013 at 3:04 PM, Adam Young <span dir="ltr"><<a href="mailto:ayoung@redhat.com" target="_blank">ayoung@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 03/26/2013 01:34 PM, David Kranz wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This is without memcache in auth_token. I was trying to find a way past <a href="https://bugs.launchpad.net/keystone/+bug/1020127" target="_blank">https://bugs.launchpad.net/<u></u>keystone/+bug/1020127</a><br>
which I think I now have. I  would appreciate it if you could validate my comment at the end of that ticket. Here, I just thought that the keystone<br>
throughput was very low. I know that swift should not be hitting it so hard. If you were referring to using memcache in the keystone server itself then<br>
</blockquote></div>
You can use memcached as an alternate token  back end, but I have no reason to thin it would perform any better than SQL.  It was broken until fairly recently, too, so I suspect it is not used much in the wild.<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I didn't know you could do that.<br>
<br>
 -David<br>
<br>
<br>
<br>
On 3/26/2013 12:33 PM, Chmouel Boudjnah wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
this seems to be pretty low, do you have memcaching enabled?<br>
<br>
On Tue, Mar 26, 2013 at 4:20 PM, David Kranz <<a href="mailto:david.kranz@qrclab.com" target="_blank">david.kranz@qrclab.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Related to this, I measured that the rate at which keystone (running on a<br>
real fairly hefty server) can handle the requests coming from the auth_token<br>
middleware (no pki tokens) is about 16/s. That seems pretty low to me. Is<br>
there some other keystone performance problem here, or is that not<br>
surprising?<br>
<br>
  -David<br>
<br>
<br>
On 3/24/2013 9:11 PM, Jay Pipes wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sure, you could do that, of course. Just like you could use gunicorn or<br>
some other web server. Just like you could deploy any of the other<br>
OpenStack services that way.<br>
<br>
It would just be nice if one could configure Keystone in the same manner<br>
that all the other OpenStack services are configured.<br>
<br>
-jay<br>
<br>
On 03/23/2013 01:19 PM, Joshua Harlow wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
See: <a href="https://github.com/openstack/keystone/tree/master/httpd" target="_blank">https://github.com/openstack/<u></u>keystone/tree/master/httpd</a><br>
<br>
For example...<br>
<br>
This lets apache do the multiprocess instead of how nova, glance ...<br>
have basically recreated the same mechanism that apache has had for<br>
years.<br>
<br>
Sent from my really tiny device...<br>
<br>
On Mar 23, 2013, at 10:14 AM, "Joshua Harlow" <<a href="mailto:harlowja@yahoo-inc.com" target="_blank">harlowja@yahoo-inc.com</a><br>
<mailto:<a href="mailto:harlowja@yahoo-inc.com" target="_blank">harlowja@yahoo-inc.com</a><u></u>>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Or I think u can run keystone in wsgi+apache easily, thus getting u the<br>
multiprocess support via apache worker processes.<br>
<br>
Sent from my really tiny<br>
device....<br>
<br>
On Mar 22, 2013, at 10:47 AM, "Jay Pipes" <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Unfortunately, Keystone's WSGI server is only a single process,<br>
</blockquote>
with a<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
greenthread pool. Unlike Glance, Nova, Cinder, and Swift, which all<br>
</blockquote>
use<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
multi-process, greenthread-pool-per-process WSGI servers[1],<br>
</blockquote>
Keystone<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
does it differently[2].<br>
<br>
There was a patchset[3] that added<br>
</blockquote>
multiprocess support to Keystone, but<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
due to objections from termie and<br>
</blockquote>
others about it not being necessary,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
it died on the vine. Termie even<br>
</blockquote>
noted that Keystone "was designed to be<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
run as multiple instances and load<br>
</blockquote>
balanced over and [he felt] that<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
should be the preferred scaling point".<br>
<br>
Because the mysql client connection is C-based, calls to it will be<br>
<br>
</blockquote>
blocking operations on greenthreads within a single process, meaning<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
even<br>
</blockquote>
if multiple greenthreads are spawned for those 200 incoming<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
requests, they<br>
</blockquote>
will be processed synchronously.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The solution is for Keystone to<br>
</blockquote>
implement the same multi-processed WSGI<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
worker stuff that is in the other<br>
</blockquote>
OpenStack projects. Or, diverge from<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
the deployment solution of Nova,<br>
</blockquote>
Glance, Cinder, and Swift, and manually<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
run multiple instances of<br>
</blockquote>
keystone, as Termie suggests.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Best,<br>
-jay<br>
<br>
[1] All pretty much<br>
</blockquote>
derived from the original Swift code, with some Oslo<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
improvements around<br>
</blockquote>
config<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[2] Compare<br>
<br>
</blockquote>
<a href="https://github.com/openstack/glance/blob/master/glance/common/wsgi.py" target="_blank">https://github.com/openstack/<u></u>glance/blob/master/glance/<u></u>common/wsgi.py</a> <br>
with<br>
<br>
<a href="https://github.com/openstack/keystone/blob/master/keystone/common/wsgi.py" target="_blank">https://github.com/openstack/<u></u>keystone/blob/master/keystone/<u></u>common/wsgi.py</a> <br>
[3] <a href="https://review.openstack.org/#/c/7017/" target="_blank">https://review.openstack.org/#<u></u>/c/7017/</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 03/21/2013 07:45 AM,<br>
</blockquote>
Kanade, Rohan wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I was trying to create 200 users using<br>
</blockquote></blockquote>
the keystone client. All the<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
users are unique and are created on separate<br>
</blockquote></blockquote>
threads which are started<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
at the same time.<br>
<br>
keystone is handling<br>
</blockquote></blockquote>
each request synchronously , i.e. user 1 is<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
created, then user 2 is<br>
</blockquote></blockquote>
created ...<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Shouldnt  keystone be running a greenthread for each<br>
</blockquote></blockquote>
request and try to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
create these users asynchronously?<br>
like start<br>
</blockquote></blockquote>
creating user 1 , while handling that request, start creating<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
user 2 or<br>
</blockquote></blockquote>
user n...<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have attached the keystone service logs for further<br>
</blockquote></blockquote>
assistance.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://paste.openstack.org/show/34216/" target="_blank">http://paste.openstack.org/<u></u>show/34216/</a><br>
<br>
<br>
</blockquote></blockquote>
______________________________<u></u>______________________________<u></u>__________ <br>
Disclaimer:This email and any attachments are sent in strictest<br>
confidence for the sole use of the addressee and may contain legally<br>
privileged, confidential, and proprietary data. If you are not the<br>
intended recipient, please advise the sender by replying promptly to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
this<br>
</blockquote></blockquote>
email and then delete and destroy this email and any attachments<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
without<br>
</blockquote></blockquote>
any further use, copying or forwarding<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
</blockquote></blockquote>
______________________________<u></u>_________________<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
OpenStack-dev mailing<br>
</blockquote></blockquote>
list<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<br>
</blockquote></blockquote>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
</blockquote>
______________________________<u></u>_________________<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
OpenStack-dev mailing<br>
</blockquote>
list<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<br>
</blockquote>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
<br>
</blockquote>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</blockquote>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div>