[openstack-dev] [nova] Mysql db connection leaking?

Jay Pipes jaypipes at gmail.com
Fri Apr 17 14:50:32 UTC 2015

On 04/17/2015 05:26 AM, Qiming Teng wrote:
> On Thu, Apr 16, 2015 at 01:27:51PM -0400, Jay Pipes wrote:
>> On 04/16/2015 09:54 AM, Sean Dague wrote:
>>> On 04/16/2015 05:20 PM, Qiming Teng wrote:
>>>> Wondering if there is something misconfigured in my devstack
>>>> environment, which was reinstalled on RHEL7 about 10 days ago.
>>>> I'm often running into mysql connections problem as shown below:
>>>> $ mysql
>>>> ERROR 1040 (HY000): Too many connections
>>>> When I try dump the mysql connection list, I'm getting the followng
>>>> result after a 'systemctl restart mariadb.service':
>>>> $ mysqladmin processlist | grep nova | wc -l
>>>> 125
>>>> Most of the connections are at Sleep status:
>>>> $ mysqladmin processlist | grep nova | grep Sleep | wc -l
>>>> 123
>>>> As for the workload, I'm currently only running two VMs in a multi-host
>>>> devstack environment.
>>>> So, my questions:
>>>>    - Why do we have so many mysql connections from nova?
>>>>    - Is it possible this is caused some misconfigurations?
>>>>    - 125 connections in such a toy setup is insane, any hints on nailing
>>>>      down the connections to the responsible nova components?
>>>> Thanks.
>>>> Regards,
>>>>    Qiming
>>> No, that's about right. It's 1 connection per worker. By default most
>>> daemons start 1 worker per processor. Each OpenStack service has a bunch
>>> of daemons. It all adds up pretty quick.
>> And just to add to what Sean says above, there's nothing inherently
>> wrong with sleeping connections to MySQL. What *is* wrong, however,
>> is that the default max_connections setting in my.cnf is 150. :( I
>> frequently recommend upping that to 2000 or more on any modern
>> hardware or decent sized VM.
>> Best,
>> -jay
> Thanks, guys. So the key takeaway for me is:
>   - 100~200 mysql connections is not big problem, provided those
>     connections are sleeping;

Yep, correct, for MySQL.

>   - Tuning mysql to support larger number of user connections is a
>     must;

Yes, because the default is a measly 150 max_connections.

>   - Number of mysql connections is not proportional to the number of
>     VMs, it is more related to the number of cores, number of workers
>     etc.

Yes. nova-compute workers (which are on each compute host that houses 
VMs) actually do not contact the database directly. Instead, they 
communicate with the nova-conductor service, which itself communicates 
with the database. The nova-conductor service by default will maintain a 
number of processes equal to the number of cores on the VM or baremetal 
machine. Each of these processes will keep a pool of connections to MySQL.

In addition to nova-conductor, there are various other Nova services 
that also make connections to MySQL.


More information about the OpenStack-dev mailing list