Open Stack

On 6/18/15 3:53 PM, Sean Dague wrote:
> On 06/18/2015 03:47 PM, Mike Bayer wrote:
>>
>> On 6/18/15 1:28 PM, Matt Riedemann wrote:
>>>
>>> It's not only neutron, I saw some pymysql failures in nova the other
>>> day for 'too many connections' or some such related error.
>> "too many connections" is an error raised by MySQL when more connections
>> are attempting to connect than the max_connections setting.  It defaults
>> to 100 but it is often set at 1024 in distributions.    An Openstack
>> application that uses oslo.db will go through oslo.db's create_engine()
>> routine which defaults the max number of connections per python process
>> to 15, and when the process is idle, this drops to 5.
>>
>> Which means, there have to be over 60 python processes all hitting the
>> DB at the max to get that error, if max_connections is at 1024.    Or
>> the connection pool is totally broken (e.g. with eventlet monkeypatching
>> perhaps), which would be a critical issue for me to fix.
>>
>> So I'd love to see how the "too many connections" error actually
>> happens.  Please share whatever you have on that.
> I believe the distro defaults in Ubuntu are the mysql defaults, which is
> 151 - https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html

We've been watching a "too many connections" issue internally and I had 
assumed our connections were at 100 or 150, but when I poked around i 
saw that RHEL is setting this at 1024 for mariadb-galera at least.    So 
it's worth knowing what the number is if we see this actually happen 
somewhere.  Quickest way to see it is to just query for it against MySQL 
itself with "show variables like 'max_connections';".

>
> If we're allowing 15 connections per python process, and the APIs are
> running in a worker mode, which means 8 workers per API server, yes,
> that's going to add up really quick.
>
> Perhaps under these scenarios we need to raise this to 1024 in devstack?

yes, that would be appropriate.

>
> 	-Sean
>