Open Stack

+openstack-dev since I believe this is an issue with the Heat source code.

On 06/18/2018 11:19 AM, Spyros Trigazis wrote:
> Hello list,
> 
> I'm hitting quite easily this [1] exception with heat. The db server is 
> configured to have 1000
> max_connnections and 1000 max_user_connections and in the database 
> section of heat
> conf I have these values set:
> max_pool_size = 22
> max_overflow = 0
> Full config attached.
> 
> I ended up with this configuration based on this formula:
> num_heat_hosts=4
> heat_api_workers=2
> heat_api_cfn_workers=2
> num_engine_workers=4
> max_pool_size=22
> max_overflow=0
> num_heat_hosts * (max_pool_size + max_overflow) * (heat_api_workers + 
> num_engine_workers + heat_api_cfn_workers)
> 704
> 
> What I have noticed is that the number of connections I expected with 
> the above formula is not respected.
> Based on this formula each node (every node runs the heat-api, 
> heat-api-cfn and heat-engine) should
> use up to 176 connections but they even reach 400 connections.
> 
> Has anyone noticed a similar behavior?

Looking through the Heat code, I see that there are many methods in the 
/heat/db/sqlalchemy/api.py module that use a SQLAlchemy session but 
never actually call session.close() [1] which means that the session 
will not be released back to the connection pool, which might be the 
reason why connections keep piling up.

Not sure if there's any setting in Heat that will fix this problem. 
Disabling connection pooling will likely not help since connections are 
not properly being closed and returned to the connection pool to begin with.

Best,
-jay

[1] Heat apparently doesn't use the oslo.db enginefacade transaction 
context managers either, which would help with this problem since the 
transaction context manager would take responsibility for calling 
session.flush()/close() appropriately.

https://github.com/openstack/oslo.db/blob/43af1cf08372006aa46d836ec45482dd4b5b5349/oslo_db/sqlalchemy/enginefacade.py#L626