Thanks for your reply, Chris. Following your advice, I adjusted the values in 'nova.conf' file. Setting max_connections in the mysql config file to 4096, the modified values are: 1. max_pool_size in [database] section to 1000 2. max_pool_size in [api_database] section to 1000 3. max_overflow in [database] section to 1000 4. max_overflow in [api_database] section to 1000 However, the problem which some services, such as Nova or Neutron, cannot connect to mysql DB occurred again. So that, I also modified the number of workers you had mentioned as follows: 1. osapi_compute_workers in [DEFAULT] section to 128 2. metadata_workers in [DEFAULT] section to 128 3. workers in [conductor] section to 128 And then, I could not use all openstack command at all. I mean, all openstack command did not work with "An unexpected error prevented the server from fulfilling your request. (HTTP 500)" error messages. After looking into some config files of the openstack services, I found out some similar parameters with the parameters you had mentioned were in the config files. For example, in neutron.conf file, max_pool_size also exists and glance-api.conf has worker parameter, either. I'm confused for what parameters should I use. Actually, I'm not sure of functions of the parameters and I need your help more. Thanks! Regards, On Sat, Dec 29, 2018 at 2:11 PM Chris Hoge <chris@openstack.org> wrote:

I’ve run into something similar. By default there are a few things going on with Nova. To begin with, Nova starts up several services multiplied by the number of cores on your system. When you add to this that Nova doesn't by default constrain its database connection pool and it's eager to create new database connections, you can quickly open tens of thousands of new connections across more than a dozen processes in a matter of minutes when starting a complete Nova control plane.

Thankfully, this is easy to fix by setting the max_pool_size and max_overflow parameters, as well as constraining the number of workers for the different processes in nova.conf.

You’ll want to tune your mysql max-connections to match your expected volume of connections from the control plane.

Hope that helps.

On Dec 28, 2018, at 8:26 AM, Minjun Hong <nickeysgo@gmail.com> wrote:

Hi. I'm currently trying to substitute the existing controller node with a new server. After I install the 'Minimal deployment for Queens' on the new server, I connect all nodes.

However, when I made an instance on Horizon, there were some error messages, as below link:

https://imgur.com/hSfHx4M

And, when I typed 'openstack flavor list', the result was as follows:

...

# openstack flavor list Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-666d4e30-081a-429f-88ae-763e62990770)

Moreover, I found something regarding this problem in the Nova log:

2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi

...

[req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')

2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi Traceback (most

...

recent call last): 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 50, in detail 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limited_flavors = self._get_flavors(req) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 117, in _get_flavors 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limit=limit, marker=marker) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/objects/flavor.py", line 650, in get_all 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi marker=marker) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 985, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi with self._transaction_scope(context): 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi context=context) as resource: 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi bind=self.connection, mode=self.mode) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._start() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi engine_args, maker_args) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi sql_connection=sql_connection, **engine_kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in wrapper 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi test_conn = _test_connection(engine, max_retries, retry_interval) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return engine.connect() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2091, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._connection_cls(self, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 90, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi if connection is not None else engine.raw_connection() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.pool.unique_connection, _connection) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766, in _checkout 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi fairy = _ConnectionRecord.checkout(pool) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516, in checkout 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi rec = pool._do_get() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138, in _do_get 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._dec_overflow() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi compat.reraise(exc_type, exc_value, exc_tb) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135, in _do_get 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._create_connection() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333, in _create_connection 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionRecord(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.__connect(first_connect_check=True) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651, in __connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi connection = pool._invoke_creator(self) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 105, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return dialect.connect(*cargs, **cparams) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 393, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.dbapi.connect(*cargs, **cparams) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return Connection(*args, **kwargs) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in __init__ 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.connect() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in connect 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._request_authentication() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in _request_authentication 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi auth_packet = self._read_packet() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi packet.check_error() 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi raise errorclass(errno, errval) 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') 2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi 2018-12-29 00:58:52.930 3765 INFO nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> 2018-12-29 00:58:52.931 3765 INFO nova.osapi_compute.wsgi.server [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] 10.150.21.195 "GET /v2.1/flavors/detail HTTP/1.1" status: 500 len: 638 time: 0.1569352

I already did googling with 'Too many connections' but, there were only results about 'max_connections' of the mysql config file. I also set the value as 4096 and I think it is enough.

Now, I do not know what to look for to resolve this problem. Please give me any clue.

Thanks! Regards,