Thanks for your reply, Chris.
Following your advice, I adjusted the values in 'nova.conf' file.
Setting max_connections in the mysql config file to 4096, the modified values are:

1. max_pool_size in [database] section  to 1000
2. max_pool_size in [api_database] section to 1000
3. max_overflow in [database] section to 1000
4. max_overflow in [api_database] section to 1000

However, the problem which some services, such as Nova or Neutron, cannot connect to mysql DB occurred again.
So that, I also modified the number of workers you had mentioned as follows:

1. osapi_compute_workers in [DEFAULT] section to 128
2. metadata_workers in [DEFAULT] section to 128
3. workers in [conductor] section to 128

And then, I could not use all openstack command at all. I mean, all openstack command did not work with "An unexpected error prevented the server from fulfilling your request. (HTTP 500)" error messages.

After looking into some config files of the openstack services, I found out some similar parameters with the parameters you had mentioned were in the config files.
For example, in neutron.conf file, max_pool_size also exists and glance-api.conf has worker parameter, either.

I'm confused for what parameters should I use.
Actually, I'm not sure of functions of the parameters and I need your help more.

Thanks!
Regards,

On Sat, Dec 29, 2018 at 2:11 PM Chris Hoge <chris@openstack.org> wrote:
I’ve run into something similar. By default there are a few things going
on with Nova. To begin with, Nova starts up several services multiplied
by the number of cores on your system. When you add to this that Nova
doesn't by default constrain its database connection pool and it's eager
to create new database connections, you can quickly open tens of
thousands of new connections across more than a dozen processes in a
matter of minutes when starting a complete Nova control plane.

Thankfully, this is easy to fix by setting the max_pool_size and
max_overflow parameters, as well as constraining the number of workers
for the different processes in nova.conf.

You’ll want to tune your mysql max-connections to match your expected
volume of connections from the control plane.

Hope that helps.

On Dec 28, 2018, at 8:26 AM, Minjun Hong <nickeysgo@gmail.com> wrote:

Hi.
I'm currently trying to substitute the existing controller node with a new server.
After I install the 'Minimal deployment for Queens' on the new server, I connect all nodes.

However, when I made an instance on Horizon, there were some error messages, as below link:


And, when I typed 'openstack flavor list', the result was as follows:
# openstack flavor list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-666d4e30-081a-429f-88ae-763e62990770)

Moreover, I found something regarding this problem in the Nova log:

2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 50, in detail
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     limited_flavors = self._get_flavors(req)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 117, in _get_flavors
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     limit=limit, marker=marker)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/nova/objects/flavor.py", line 650, in get_all
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     marker=marker)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 985, in wrapper
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     with self._transaction_scope(context):
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return self.gen.next()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     context=context) as resource:
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return self.gen.next()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     bind=self.connection, mode=self.mode)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self._start()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     engine_args, maker_args)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     sql_connection=sql_connection, **engine_kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in wrapper
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     test_conn = _test_connection(engine, max_retries, retry_interval)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return engine.connect()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2091, in connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return self._connection_cls(self, **kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 90, in __init__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     if connection is not None else engine.raw_connection()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self.pool.unique_connection, _connection)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     e, dialect, self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     util.raise_from_cause(newraise, exc_info)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return fn()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return _ConnectionFairy._checkout(self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     e, dialect, self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     util.raise_from_cause(newraise, exc_info)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return fn()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return _ConnectionFairy._checkout(self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766, in _checkout
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     fairy = _ConnectionRecord.checkout(pool)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516, in checkout
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     rec = pool._do_get()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138, in _do_get
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self._dec_overflow()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     compat.reraise(exc_type, exc_value, exc_tb)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135, in _do_get
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return self._create_connection()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333, in _create_connection
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return _ConnectionRecord(self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461, in __init__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self.__connect(first_connect_check=True)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651, in __connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     connection = pool._invoke_creator(self)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return dialect.connect(*cargs, **cparams)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 393, in connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return self.dbapi.connect(*cargs, **cparams)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     return Connection(*args, **kwargs)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in __init__
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self.connect()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in connect
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     self._request_authentication()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in _request_authentication
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     auth_packet = self._read_packet()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     packet.check_error()
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi
2018-12-29 00:58:52.930 3765 INFO nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2018-12-29 00:58:52.931 3765 INFO nova.osapi_compute.wsgi.server [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] 10.150.21.195 "GET /v2.1/flavors/detail HTTP/1.1" status: 500 len: 638 time: 0.1569352

I already did googling with 'Too many connections' but, there were only results about 'max_connections' of the mysql config file.
I also set the value as 4096 and I think it is enough.

Now, I do not know what to look for to resolve this problem.
Please give me any clue.

Thanks!
Regards,