[Openstack-operators] [openstack-dev] [openstack-operators][heat][oslo.db][magnum] Configure maximum number of db connections

Spyros Trigazis strigazi at gmail.com
Tue Jun 19 09:17:58 UTC 2018


Hello lists,

With heat's team help I figured it out. Thanks Jay for looking into it.

The issue is coming from [1], where the max_overflow is set to
 executor_thread_pool_size if it is set to a lower value to address
another issue. In my case, I had a lot of RAM and CPU so I could
push for threads but I was "short" in db connections. The formula to
calculate the number of connections can be like this:
num_heat_hosts=4
heat_api_workers=2
heat_api_cfn_workers=2
num_engine_workers=4
executor_thread_pool_size = 22
max_pool_size=4
max_overflow=executor_thread_pool_size
num_heat_hosts * (max_pool_size + max_overflow) * (heat_api_workers +
num_engine_workers + heat_api_cfn_workers)
832

And  a note for magnum deployments medium to large, see the options
we have changed in heat conf and change according to your needs.
The db configuration described here and changes we discovered in a
previous scale test can help to have a stable magnum and heat service.

For large stacks or projects with many stacks you need to change
the following in these values or better, according to your needs.

[Default]
executor_thread_pool_size = 22
max_resources_per_stack = -1
max_stacks_per_tenant = 10000
action_retry_limit = 10
client_retry_limit = 10
engine_life_check_timeout = 600
max_template_size = 5242880
rpc_poll_timeout = 600
rpc_response_timeout = 600
num_engine_workers = 4

[database]
max_pool_size = 4
max_overflow = 22
Cheers,
Spyros

[heat_api]

workers = 2

[heat_api_cfn]
workers = 2

Cheers,
Spyros

ps We will update the magnum docs as well

[1]
http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/service.py#n375


On Mon, 18 Jun 2018 at 19:39, Jay Pipes <jaypipes at gmail.com> wrote:

> +openstack-dev since I believe this is an issue with the Heat source code.
>
> On 06/18/2018 11:19 AM, Spyros Trigazis wrote:
> > Hello list,
> >
> > I'm hitting quite easily this [1] exception with heat. The db server is
> > configured to have 1000
> > max_connnections and 1000 max_user_connections and in the database
> > section of heat
> > conf I have these values set:
> > max_pool_size = 22
> > max_overflow = 0
> > Full config attached.
> >
> > I ended up with this configuration based on this formula:
> > num_heat_hosts=4
> > heat_api_workers=2
> > heat_api_cfn_workers=2
> > num_engine_workers=4
> > max_pool_size=22
> > max_overflow=0
> > num_heat_hosts * (max_pool_size + max_overflow) * (heat_api_workers +
> > num_engine_workers + heat_api_cfn_workers)
> > 704
> >
> > What I have noticed is that the number of connections I expected with
> > the above formula is not respected.
> > Based on this formula each node (every node runs the heat-api,
> > heat-api-cfn and heat-engine) should
> > use up to 176 connections but they even reach 400 connections.
> >
> > Has anyone noticed a similar behavior?
>
> Looking through the Heat code, I see that there are many methods in the
> /heat/db/sqlalchemy/api.py module that use a SQLAlchemy session but
> never actually call session.close() [1] which means that the session
> will not be released back to the connection pool, which might be the
> reason why connections keep piling up.
>
> Not sure if there's any setting in Heat that will fix this problem.
> Disabling connection pooling will likely not help since connections are
> not properly being closed and returned to the connection pool to begin
> with.
>
> Best,
> -jay
>
> [1] Heat apparently doesn't use the oslo.db enginefacade transaction
> context managers either, which would help with this problem since the
> transaction context manager would take responsibility for calling
> session.flush()/close() appropriately.
>
>
> https://github.com/openstack/oslo.db/blob/43af1cf08372006aa46d836ec45482dd4b5b5349/oslo_db/sqlalchemy/enginefacade.py#L626
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180619/5491aeff/attachment.html>


More information about the OpenStack-operators mailing list