Open Stack

On 9/18/2014 12:35 AM, Morgan Fainberg wrote:
> -----Original Message-----
> From: Dean Troyer <dtroyer at gmail.com>
> Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>>
> Date: September 17, 2014 at 21:21:47
> To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>>
> Subject:  Re: [openstack-dev] PostgreSQL jobs slow in the gate
>
>>>
>>> Clark Boylan Wrotr :
>>>
>>>> It appears that keystone eventlet is the source of the slowness in this
>>>> job. With keystone eventlet all of the devstack jobs are slower and with
>>>> keystone wsgi all of the jobs are quicker. Probably need to collect a
>>>> bit more data but this doesn't look good for keystone eventlet.
>>>>
>>>
>> On Wed, Sep 17, 2014 at 11:02 PM, Morgan Fainberg <
>> morgan.fainberg at gmail.com> wrote:
>>
>>> I've kicked off a test[1] as well to check into some tunable options
>>> (eventlet workers) for improving keystone eventlet performance. I'll circle
>>> back with the infra team once we have a little more data on both fronts.
>>> The Keystone team will use this data to figure out the best way to approach
>>> this issue.
>>>
>>
>> Brant submitted https://review.openstack.org/#/c/121384/ to up the Keystone
>> workers when API_WORKERS is set.
>>
>> I submitted https://review.openstack.org/#/c/122013/ to set a scaling
>> default for API_WORKERS based on the available CPUs ((nproc+1)/2). There
>> is a summary in that commit message of the current reviews addressing the
>> workers in various projects.
>>
>> I think it has become clear that DevStack needs to set a default for most
>> services that are currently either too big (nproc) or too small (Keystone
>> at 1). Of course, moving things to mod_wsgi moots all of that, but it'll
>> be a while before everything moves.
>>
>> dt
>>
>> --
>>
>> Dean Troyer
>> dtroyer at gmail.com
>
> Dean,
>
> We should probably look at increasing the default number of workers as well in the Keystone configuration (rather than just devstack). It looks like, with limited datasets, we are seeing a real improvement with Keystone and 4 workers (from my previously mentioned quick test). Thanks for the extra data points. This helps to confirm the issue is Keystone under eventlet.
>
> Cheers,
> Morgan
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I pointed this out in Brant's devstack patch but we had a product team 
internally bring up this same point in Icehouse, they were really 
limited due to the eventlet workers issue in Keystone and once we 
provided the option (backported) it increased their throughput by 20%. 
We've been running with that in our internal Tempest runs (setting 
workers equal to number of CPUs / 2) and so far so good.

-- 

Thanks,

Matt Riedemann