[openstack-dev] [all] DevStack switching from MySQL-python to PyMySQL
sean at dague.net
Thu Jun 18 20:31:44 UTC 2015
On 06/18/2015 04:28 PM, Clint Byrum wrote:
> Excerpts from Sean Dague's message of 2015-06-18 12:53:16 -0700:
>> On 06/18/2015 03:47 PM, Mike Bayer wrote:
>>> On 6/18/15 1:28 PM, Matt Riedemann wrote:
>>>> It's not only neutron, I saw some pymysql failures in nova the other
>>>> day for 'too many connections' or some such related error.
>>> "too many connections" is an error raised by MySQL when more connections
>>> are attempting to connect than the max_connections setting. It defaults
>>> to 100 but it is often set at 1024 in distributions. An Openstack
>>> application that uses oslo.db will go through oslo.db's create_engine()
>>> routine which defaults the max number of connections per python process
>>> to 15, and when the process is idle, this drops to 5.
>>> Which means, there have to be over 60 python processes all hitting the
>>> DB at the max to get that error, if max_connections is at 1024. Or
>>> the connection pool is totally broken (e.g. with eventlet monkeypatching
>>> perhaps), which would be a critical issue for me to fix.
>>> So I'd love to see how the "too many connections" error actually
>>> happens. Please share whatever you have on that.
>> I believe the distro defaults in Ubuntu are the mysql defaults, which is
>> 151 - https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html
>> If we're allowing 15 connections per python process, and the APIs are
>> running in a worker mode, which means 8 workers per API server, yes,
>> that's going to add up really quick.
>> Perhaps under these scenarios we need to raise this to 1024 in devstack?
> Indeed, mysqld needs tuning for better concurrency. I submitted this
> which should help avoid the low connection count and also will
> unserialize read/writes on a single table which should lead to tests
> doing multiple operations asynchronously finishing a bit quicker.
> Note that we MIGHT want to raise the innodb_buffer_pool_size. The
> default since 5.5 has been 128MB (It used to be _8MB_). However, I
> imagine memory can get tight on devstack nodes and laptops, so it's also
> perfectly valid to keep it at 128MB and just let operations go to disk
> more often.
Yeh, we are trying to stay useful on 4G VMs, we did turn off some
default services for that reason. So lets not up memory unless we have to.
More information about the OpenStack-dev