[openstack-dev] Fwd: [keystone] Keystone handling http requests synchronously

Chmouel Boudjnah chmouel at chmouel.com
Fri Mar 29 09:34:24 UTC 2013


FYI


---------- Forwarded message ----------
From: Adam Young <ayoung at redhat.com>
Date: Thu, Mar 28, 2013 at 10:04 PM
Subject: Re: [openstack-dev] [keystone] Keystone handling http
requests synchronously
To: openstack-dev at lists.openstack.org


On 03/26/2013 01:34 PM, David Kranz wrote:
>
> This is without memcache in auth_token. I was trying to find a way past https://bugs.launchpad.net/keystone/+bug/1020127
> which I think I now have. I  would appreciate it if you could validate my comment at the end of that ticket. Here, I just thought that the keystone
> throughput was very low. I know that swift should not be hitting it so hard. If you were referring to using memcache in the keystone server itself then

You can use memcached as an alternate token  back end, but I have no
reason to thin it would perform any better than SQL.  It was broken
until fairly recently, too, so I suspect it is not used much in the
wild.



> I didn't know you could do that.
>
>  -David
>
>
>
> On 3/26/2013 12:33 PM, Chmouel Boudjnah wrote:
>>
>> this seems to be pretty low, do you have memcaching enabled?
>>
>> On Tue, Mar 26, 2013 at 4:20 PM, David Kranz <david.kranz at qrclab.com> wrote:
>>>
>>> Related to this, I measured that the rate at which keystone (running on a
>>> real fairly hefty server) can handle the requests coming from the auth_token
>>> middleware (no pki tokens) is about 16/s. That seems pretty low to me. Is
>>> there some other keystone performance problem here, or is that not
>>> surprising?
>>>
>>>   -David
>>>
>>>
>>> On 3/24/2013 9:11 PM, Jay Pipes wrote:
>>>>
>>>> Sure, you could do that, of course. Just like you could use gunicorn or
>>>> some other web server. Just like you could deploy any of the other
>>>> OpenStack services that way.
>>>>
>>>> It would just be nice if one could configure Keystone in the same manner
>>>> that all the other OpenStack services are configured.
>>>>
>>>> -jay
>>>>
>>>> On 03/23/2013 01:19 PM, Joshua Harlow wrote:
>>>>>
>>>>> See: https://github.com/openstack/keystone/tree/master/httpd
>>>>>
>>>>> For example...
>>>>>
>>>>> This lets apache do the multiprocess instead of how nova, glance ...
>>>>> have basically recreated the same mechanism that apache has had for
>>>>> years.
>>>>>
>>>>> Sent from my really tiny device...
>>>>>
>>>>> On Mar 23, 2013, at 10:14 AM, "Joshua Harlow" <harlowja at yahoo-inc.com
>>>>> <mailto:harlowja at yahoo-inc.com>> wrote:
>>>>>
>>>>>> Or I think u can run keystone in wsgi+apache easily, thus getting u the
>>>>>> multiprocess support via apache worker processes.
>>>>>>
>>>>>> Sent from my really tiny
>>>>>> device....
>>>>>>
>>>>>> On Mar 22, 2013, at 10:47 AM, "Jay Pipes" <jaypipes at gmail.com
>>>>>> <mailto:jaypipes at gmail.com>>
>>>>>> wrote:
>>>>>>
>>>>>>> Unfortunately, Keystone's WSGI server is only a single process,
>>>>>>
>>>>>> with a
>>>>>>>
>>>>>>> greenthread pool. Unlike Glance, Nova, Cinder, and Swift, which all
>>>>>>
>>>>>> use
>>>>>>>
>>>>>>> multi-process, greenthread-pool-per-process WSGI servers[1],
>>>>>>
>>>>>> Keystone
>>>>>>>
>>>>>>> does it differently[2].
>>>>>>>
>>>>>>> There was a patchset[3] that added
>>>>>>
>>>>>> multiprocess support to Keystone, but
>>>>>>>
>>>>>>> due to objections from termie and
>>>>>>
>>>>>> others about it not being necessary,
>>>>>>>
>>>>>>> it died on the vine. Termie even
>>>>>>
>>>>>> noted that Keystone "was designed to be
>>>>>>>
>>>>>>> run as multiple instances and load
>>>>>>
>>>>>> balanced over and [he felt] that
>>>>>>>
>>>>>>> should be the preferred scaling point".
>>>>>>>
>>>>>>> Because the mysql client connection is C-based, calls to it will be
>>>>>>>
>>>>>> blocking operations on greenthreads within a single process, meaning
>>>>>>>
>>>>>>> even
>>>>>>
>>>>>> if multiple greenthreads are spawned for those 200 incoming
>>>>>>>
>>>>>>> requests, they
>>>>>>
>>>>>> will be processed synchronously.
>>>>>>>
>>>>>>> The solution is for Keystone to
>>>>>>
>>>>>> implement the same multi-processed WSGI
>>>>>>>
>>>>>>> worker stuff that is in the other
>>>>>>
>>>>>> OpenStack projects. Or, diverge from
>>>>>>>
>>>>>>> the deployment solution of Nova,
>>>>>>
>>>>>> Glance, Cinder, and Swift, and manually
>>>>>>>
>>>>>>> run multiple instances of
>>>>>>
>>>>>> keystone, as Termie suggests.
>>>>>>>
>>>>>>> Best,
>>>>>>> -jay
>>>>>>>
>>>>>>> [1] All pretty much
>>>>>>
>>>>>> derived from the original Swift code, with some Oslo
>>>>>>>
>>>>>>> improvements around
>>>>>>
>>>>>> config
>>>>>>>
>>>>>>> [2] Compare
>>>>>>>
>>>>>> https://github.com/openstack/glance/blob/master/glance/common/wsgi.py
>>>>>> with
>>>>>>
>>>>>> https://github.com/openstack/keystone/blob/master/keystone/common/wsgi.py
>>>>>> [3] https://review.openstack.org/#/c/7017/
>>>>>>>
>>>>>>> On 03/21/2013 07:45 AM,
>>>>>>
>>>>>> Kanade, Rohan wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I was trying to create 200 users using
>>>>>>
>>>>>> the keystone client. All the
>>>>>>>>
>>>>>>>> users are unique and are created on separate
>>>>>>
>>>>>> threads which are started
>>>>>>>>
>>>>>>>> at the same time.
>>>>>>>>
>>>>>>>> keystone is handling
>>>>>>
>>>>>> each request synchronously , i.e. user 1 is
>>>>>>>>
>>>>>>>> created, then user 2 is
>>>>>>
>>>>>> created ...
>>>>>>>>
>>>>>>>> Shouldnt  keystone be running a greenthread for each
>>>>>>
>>>>>> request and try to
>>>>>>>>
>>>>>>>> create these users asynchronously?
>>>>>>>> like start
>>>>>>
>>>>>> creating user 1 , while handling that request, start creating
>>>>>>>>
>>>>>>>> user 2 or
>>>>>>
>>>>>> user n...
>>>>>>>>
>>>>>>>> I have attached the keystone service logs for further
>>>>>>
>>>>>> assistance.
>>>>>>>>
>>>>>>>> http://paste.openstack.org/show/34216/
>>>>>>>>
>>>>>>>>
>>>>>> ______________________________________________________________________
>>>>>> Disclaimer:This email and any attachments are sent in strictest
>>>>>> confidence for the sole use of the addressee and may contain legally
>>>>>> privileged, confidential, and proprietary data. If you are not the
>>>>>> intended recipient, please advise the sender by replying promptly to
>>>>>>>>
>>>>>>>> this
>>>>>>
>>>>>> email and then delete and destroy this email and any attachments
>>>>>>>>
>>>>>>>> without
>>>>>>
>>>>>> any further use, copying or forwarding
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> OpenStack-dev mailing
>>>>>>
>>>>>> list
>>>>>>>>
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>>>
>>>>>>> OpenStack-dev mailing
>>>>>>
>>>>>> list
>>>>>>>
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list