[openstack-discuss][Nova][Horizon] 'Too many connections' error
Hi. I'm currently trying to substitute the existing controller node with a new server. After I install the 'Minimal deployment for Queens' on the new server, I connect all nodes. However, when I made an instance on Horizon, there were some error messages, as below link: https://imgur.com/hSfHx4M And, when I typed 'openstack flavor list', the result was as follows:
# openstack flavor list Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-666d4e30-081a-429f-88ae-763e62990770)
Moreover, I found something regarding this problem in the Nova log: 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
[req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi Traceback (most
recent call last): 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 50, in detail 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limited_flavors = self._get_flavors(req) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 117, in _get_flavors 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limit=limit, marker=marker) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/objects/flavor.py", line 650, in get_all 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi marker=marker) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 985, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi with self._transaction_scope(context): 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi context=context) as resource: 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi bind=self.connection, mode=self.mode) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._start() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi engine_args, maker_args) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi sql_connection=sql_connection, **engine_kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi test_conn = _test_connection(engine, max_retries, retry_interval) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return engine.connect() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2091, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._connection_cls(self, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 90, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi if connection is not None else engine.raw_connection() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.pool.unique_connection, _connection) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766, in _checkout 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi fairy = _ConnectionRecord.checkout(pool) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516, in checkout 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi rec = pool._do_get() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138, in _do_get 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._dec_overflow() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi compat.reraise(exc_type, exc_value, exc_tb) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135, in _do_get 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._create_connection() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333, in _create_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionRecord(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.__connect(first_connect_check=True) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651, in __connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi connection = pool._invoke_creator(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 105, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return dialect.connect(*cargs, **cparams) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 393, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.dbapi.connect(*cargs, **cparams) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return Connection(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.connect() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._request_authentication() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in _request_authentication 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi auth_packet = self._read_packet() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi packet.check_error() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi raise errorclass(errno, errval) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi 2018-12-29 00:58:52.930 3765 INFO nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> 2018-12-29 00:58:52.931 3765 INFO nova.osapi_compute.wsgi.server [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] 10.150.21.195 "GET /v2.1/flavors/detail HTTP/1.1" status: 500 len: 638 time: 0.1569352
I already did googling with 'Too many connections' but, there were only results about 'max_connections' of the mysql config file. I also set the value as 4096 and I think it is enough. Now, I do not know what to look for to resolve this problem. Please give me any clue. Thanks! Regards,
participants (1)
-
Minjun Hong