Open Stack

On 04/17/2015 05:26 AM, Qiming Teng wrote:
> On Thu, Apr 16, 2015 at 01:27:51PM -0400, Jay Pipes wrote:
>> On 04/16/2015 09:54 AM, Sean Dague wrote:
>>> On 04/16/2015 05:20 PM, Qiming Teng wrote:
>>>>
>>>> Wondering if there is something misconfigured in my devstack
>>>> environment, which was reinstalled on RHEL7 about 10 days ago.
>>>> I'm often running into mysql connections problem as shown below:
>>>>
>>>> $ mysql
>>>> ERROR 1040 (HY000): Too many connections
>>>>
>>>> When I try dump the mysql connection list, I'm getting the followng
>>>> result after a 'systemctl restart mariadb.service':
>>>>
>>>> $ mysqladmin processlist | grep nova | wc -l
>>>> 125
>>>>
>>>> Most of the connections are at Sleep status:
>>>>
>>>> $ mysqladmin processlist | grep nova | grep Sleep | wc -l
>>>> 123
>>>>
>>>> As for the workload, I'm currently only running two VMs in a multi-host
>>>> devstack environment.
>>>>
>>>> So, my questions:
>>>>
>>>>    - Why do we have so many mysql connections from nova?
>>>>    - Is it possible this is caused some misconfigurations?
>>>>    - 125 connections in such a toy setup is insane, any hints on nailing
>>>>      down the connections to the responsible nova components?
>>>>
>>>> Thanks.
>>>>
>>>> Regards,
>>>>    Qiming
>>>
>>> No, that's about right. It's 1 connection per worker. By default most
>>> daemons start 1 worker per processor. Each OpenStack service has a bunch
>>> of daemons. It all adds up pretty quick.
>>
>> And just to add to what Sean says above, there's nothing inherently
>> wrong with sleeping connections to MySQL. What *is* wrong, however,
>> is that the default max_connections setting in my.cnf is 150. :( I
>> frequently recommend upping that to 2000 or more on any modern
>> hardware or decent sized VM.
>>
>> Best,
>> -jay
>>
> Thanks, guys. So the key takeaway for me is:
>
>   - 100~200 mysql connections is not big problem, provided those
>     connections are sleeping;

Yep, correct, for MySQL.

>   - Tuning mysql to support larger number of user connections is a
>     must;

Yes, because the default is a measly 150 max_connections.

>   - Number of mysql connections is not proportional to the number of
>     VMs, it is more related to the number of cores, number of workers
>     etc.

Yes. nova-compute workers (which are on each compute host that houses 
VMs) actually do not contact the database directly. Instead, they 
communicate with the nova-conductor service, which itself communicates 
with the database. The nova-conductor service by default will maintain a 
number of processes equal to the number of cores on the VM or baremetal 
machine. Each of these processes will keep a pool of connections to MySQL.

In addition to nova-conductor, there are various other Nova services 
that also make connections to MySQL.

Best,
-jay