[openstack-discuss][Nova][Horizon] 'Too many connections' error
Minjun Hong
nickeysgo at gmail.com
Sun Dec 30 09:41:35 UTC 2018
Thanks for your reply, Chris.
Following your advice, I adjusted the values in 'nova.conf' file.
Setting max_connections in the mysql config file to 4096, the modified
values are:
1. max_pool_size in [database] section to 1000
2. max_pool_size in [api_database] section to 1000
3. max_overflow in [database] section to 1000
4. max_overflow in [api_database] section to 1000
However, the problem which some services, such as Nova or Neutron, cannot
connect to mysql DB occurred again.
So that, I also modified the number of workers you had mentioned as follows:
1. osapi_compute_workers in [DEFAULT] section to 128
2. metadata_workers in [DEFAULT] section to 128
3. workers in [conductor] section to 128
And then, I could not use all openstack command at all. I mean, all
openstack command did not work with "An unexpected error prevented the
server from fulfilling your request. (HTTP 500)" error messages.
After looking into some config files of the openstack services, I found out
some similar parameters with the parameters you had mentioned were in the
config files.
For example, in neutron.conf file, max_pool_size also exists and
glance-api.conf has worker parameter, either.
I'm confused for what parameters should I use.
Actually, I'm not sure of functions of the parameters and I need your help
more.
Thanks!
Regards,
On Sat, Dec 29, 2018 at 2:11 PM Chris Hoge <chris at openstack.org> wrote:
> I’ve run into something similar. By default there are a few things going
> on with Nova. To begin with, Nova starts up several services multiplied
> by the number of cores on your system. When you add to this that Nova
> doesn't by default constrain its database connection pool and it's eager
> to create new database connections, you can quickly open tens of
> thousands of new connections across more than a dozen processes in a
> matter of minutes when starting a complete Nova control plane.
>
> Thankfully, this is easy to fix by setting the max_pool_size and
> max_overflow parameters, as well as constraining the number of workers
> for the different processes in nova.conf.
>
> You’ll want to tune your mysql max-connections to match your expected
> volume of connections from the control plane.
>
> Hope that helps.
>
> On Dec 28, 2018, at 8:26 AM, Minjun Hong <nickeysgo at gmail.com> wrote:
>
> Hi.
> I'm currently trying to substitute the existing controller node with a new
> server.
> After I install the 'Minimal deployment for Queens' on the new server, I
> connect all nodes.
>
> However, when I made an instance on Horizon, there were some error
> messages, as below link:
>
> https://imgur.com/hSfHx4M
>
> And, when I typed 'openstack flavor list', the result was as follows:
>
>> # openstack flavor list
>> Unexpected API Error. Please report this at
>> http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
>> <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID:
>> req-666d4e30-081a-429f-88ae-763e62990770)
>
>
> Moreover, I found something regarding this problem in the Nova log:
>
> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3
>> 96ad10a59d114042b8f1ee82c438649a - default default] Unexpected exception in
>> API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too
>> many connections')
>
> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi Traceback (most
>> recent call last):
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in
>> wrapped
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> f(*args, **kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py",
>> line 50, in detail
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> limited_flavors = self._get_flavors(req)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py",
>> line 117, in _get_flavors
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> limit=limit, marker=marker)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184,
>> in wrapper
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi result =
>> fn(cls, context, *args, **kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/nova/objects/flavor.py", line 650, in
>> get_all
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> marker=marker)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 985, in wrapper
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi with
>> self._transaction_scope(context):
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> self.gen.next()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 1035, in _transaction_scope
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> context=context) as resource:
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> self.gen.next()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 638, in _session
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> bind=self.connection, mode=self.mode)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 403, in _create_session
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self._start()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 489, in _start
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> engine_args, maker_args)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
>> 513, in _setup_for_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> sql_connection=sql_connection, **engine_kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in
>> wrapper
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> f(*args, **kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184,
>> in create_engine
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi test_conn
>> = _test_connection(engine, max_retries, retry_interval)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 362,
>> in _test_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> engine.connect()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2091, in
>> connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> self._connection_cls(self, **kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 90, in
>> __init__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi if
>> connection is not None else engine.raw_connection()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2177, in
>> raw_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self.pool.unique_connection, _connection)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in
>> _wrap_pool_connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e,
>> dialect, self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in
>> _handle_dbapi_exception_noconnection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> util.raise_from_cause(newraise, exc_info)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in
>> raise_from_cause
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> reraise(type(exception), exception, tb=exc_tb, cause=cause)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in
>> _wrap_pool_connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in
>> unique_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> _ConnectionFairy._checkout(self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in
>> _wrap_pool_connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e,
>> dialect, self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in
>> _handle_dbapi_exception_noconnection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> util.raise_from_cause(newraise, exc_info)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in
>> raise_from_cause
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> reraise(type(exception), exception, tb=exc_tb, cause=cause)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in
>> _wrap_pool_connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in
>> unique_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> _ConnectionFairy._checkout(self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766, in
>> _checkout
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi fairy =
>> _ConnectionRecord.checkout(pool)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516, in checkout
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi rec =
>> pool._do_get()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138, in _do_get
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self._dec_overflow()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66,
>> in __exit__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> compat.reraise(exc_type, exc_value, exc_tb)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135, in _do_get
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> self._create_connection()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333, in
>> _create_connection
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> _ConnectionRecord(self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461, in __init__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self.__connect(first_connect_check=True)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651, in
>> __connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi connection
>> = pool._invoke_creator(self)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line
>> 105, in connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> dialect.connect(*cargs, **cparams)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 393,
>> in connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> self.dbapi.connect(*cargs, **cparams)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return
>> Connection(*args, **kwargs)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in
>> __init__
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self.connect()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in
>> connect
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> self._request_authentication()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in
>> _request_authentication
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> auth_packet = self._read_packet()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in
>> _read_packet
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> packet.check_error()
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in
>> check_error
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> err.raise_mysql_exception(self._data)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File
>> "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in
>> raise_mysql_exception
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi raise
>> errorclass(errno, errval)
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> OperationalError: (pymysql.err.OperationalError) (1040, u'Too many
>> connections')
>> 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
>> 2018-12-29 00:58:52.930 3765 INFO nova.api.openstack.wsgi
>> [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3
>> 96ad10a59d114042b8f1ee82c438649a - default default] HTTP exception thrown:
>> Unexpected API Error. Please report this at
>> http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
>> <class 'sqlalchemy.exc.OperationalError'>
>> 2018-12-29 00:58:52.931 3765 INFO nova.osapi_compute.wsgi.server
>> [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3
>> 96ad10a59d114042b8f1ee82c438649a - default default] 10.150.21.195 "GET
>> /v2.1/flavors/detail HTTP/1.1" status: 500 len: 638 time: 0.1569352
>
>
> I already did googling with 'Too many connections' but, there were only
> results about 'max_connections' of the mysql config file.
> I also set the value as 4096 and I think it is enough.
>
> Now, I do not know what to look for to resolve this problem.
> Please give me any clue.
>
> Thanks!
> Regards,
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20181230/9216b8fd/attachment-0001.html>
More information about the openstack-discuss
mailing list