Open Stack

Excerpts from Sean Dague's message of 2015-06-18 12:53:16 -0700:
> On 06/18/2015 03:47 PM, Mike Bayer wrote:
> > 
> > 
> > On 6/18/15 1:28 PM, Matt Riedemann wrote:
> >>
> >>
> >>>
> >>
> >> It's not only neutron, I saw some pymysql failures in nova the other
> >> day for 'too many connections' or some such related error.
> > 
> > "too many connections" is an error raised by MySQL when more connections
> > are attempting to connect than the max_connections setting.  It defaults
> > to 100 but it is often set at 1024 in distributions.    An Openstack
> > application that uses oslo.db will go through oslo.db's create_engine()
> > routine which defaults the max number of connections per python process
> > to 15, and when the process is idle, this drops to 5.
> > 
> > Which means, there have to be over 60 python processes all hitting the
> > DB at the max to get that error, if max_connections is at 1024.    Or
> > the connection pool is totally broken (e.g. with eventlet monkeypatching
> > perhaps), which would be a critical issue for me to fix.
> > 
> > So I'd love to see how the "too many connections" error actually
> > happens.  Please share whatever you have on that.
> 
> I believe the distro defaults in Ubuntu are the mysql defaults, which is
> 151 - https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html
> 
> If we're allowing 15 connections per python process, and the APIs are
> running in a worker mode, which means 8 workers per API server, yes,
> that's going to add up really quick.
> 
> Perhaps under these scenarios we need to raise this to 1024 in devstack?

Indeed, mysqld needs tuning for better concurrency. I submitted this
which should help avoid the low connection count and also will
unserialize read/writes on a single table which should lead to tests
doing multiple operations asynchronously finishing a bit quicker.

https://review.openstack.org/193295

Note that we MIGHT want to raise the innodb_buffer_pool_size. The
default since 5.5 has been 128MB (It used to be _8MB_). However, I
imagine memory can get tight on devstack nodes and laptops, so it's also
perfectly valid to keep it at 128MB and just let operations go to disk
more often.