<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"></div><div class="gmail_quote"><div>Thanks for your reply, Chris.</div><div>Following your advice, I adjusted the values in 'nova.conf' file.</div><div>Setting max_connections in the mysql config file to 4096, the modified values are:</div><div><br></div><div>1. max_pool_size in [database] section to 1000</div><div>2. max_pool_size in [api_database] section to 1000</div><div>3. max_overflow in [database] section to 1000<br></div><div>4. max_overflow in [api_database] section to 1000</div><div><br></div><div>However, the problem which some services, such as Nova or Neutron, cannot connect to mysql DB occurred again.</div><div>So that, I also modified the number of workers you had mentioned as follows:<br></div><div><br></div><div>1. osapi_compute_workers in [DEFAULT] section to 128</div><div>2. metadata_workers in [DEFAULT] section to 128</div><div>3. workers in [conductor] section to 128</div><div><br></div><div>And then, I could not use all openstack command at all. I mean, all openstack command did not work with "An unexpected error prevented the server from fulfilling your request. (HTTP 500)" error messages.</div><div><br></div><div>After looking into some config files of the openstack services, I found out some similar parameters with the parameters you had mentioned were in the config files.</div><div>For example, in neutron.conf file, max_pool_size also exists and glance-api.conf has worker parameter, either.</div><div><br></div><div>I'm confused for what parameters should I use.</div><div>Actually, I'm not sure of functions of the parameters and I need your help more.</div><div><br></div><div>Thanks!</div><div>Regards,</div><div dir="ltr"><br></div><div dir="ltr">On Sat, Dec 29, 2018 at 2:11 PM Chris Hoge <<a href="mailto:chris@openstack.org">chris@openstack.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>I’ve run into something similar. By default there are a few things going</div><div>on with Nova. To begin with, Nova starts up several services multiplied</div><div>by the number of cores on your system. When you add to this that Nova</div><div>doesn't by default constrain its database connection pool and it's eager</div><div>to create new database connections, you can quickly open tens of</div><div>thousands of new connections across more than a dozen processes in a</div><div>matter of minutes when starting a complete Nova control plane.</div><div><br></div><div>Thankfully, this is easy to fix by setting the max_pool_size and</div><div>max_overflow parameters, as well as constraining the number of workers</div><div>for the different processes in nova.conf.</div><div><br></div><div>You’ll want to tune your mysql max-connections to match your expected</div><div>volume of connections from the control plane.</div><div><br></div><div>Hope that helps.</div><div><br><blockquote type="cite"><div>On Dec 28, 2018, at 8:26 AM, Minjun Hong <<a href="mailto:nickeysgo@gmail.com" target="_blank">nickeysgo@gmail.com</a>> wrote:</div><br class="gmail-m_2153596891177672072Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi.<div>I'm currently trying to substitute the existing controller node with a new server.</div><div>After I install the 'Minimal deployment for Queens' on the new server, I connect all nodes.</div><div><br></div><div>However, when I made an instance on Horizon, there were some error messages, as below link:</div><div><br></div><div><a href="https://imgur.com/hSfHx4M" target="_blank">https://imgur.com/hSfHx4M</a><br></div><div><br></div><div>And, when I typed 'openstack flavor list', the result was as follows:</div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"># openstack flavor list<br>Unexpected API Error. Please report this at <a href="http://bugs.launchpad.net/nova/" target="_blank">http://bugs.launchpad.net/nova/</a> and attach the Nova API log if possible.<br><class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-666d4e30-081a-429f-88ae-763e62990770)</blockquote></div><div><br></div><div>Moreover, I found something regarding this problem in the Nova log:<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')</blockquote><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi Traceback (most recent call last):<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 50, in detail<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limited_flavors = self._get_flavors(req)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/flavors.py", line 117, in _get_flavors<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi limit=limit, marker=marker)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/objects/flavor.py", line 650, in get_all<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi marker=marker)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 985, in wrapper<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi with self._transaction_scope(context):<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi context=context) as resource:<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.gen.next()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi bind=self.connection, mode=self.mode)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._start()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi engine_args, maker_args)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi sql_connection=sql_connection, **engine_kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in wrapper<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi test_conn = _test_connection(engine, max_retries, retry_interval)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return engine.connect()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2091, in connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._connection_cls(self, **kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 90, in __init__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi if connection is not None else engine.raw_connection()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.pool.unique_connection, _connection)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi e, dialect, self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return fn()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 328, in unique_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionFairy._checkout(self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766, in _checkout<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi fairy = _ConnectionRecord.checkout(pool)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516, in checkout<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi rec = pool._do_get()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138, in _do_get<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._dec_overflow()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi compat.reraise(exc_type, exc_value, exc_tb)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135, in _do_get<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self._create_connection()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333, in _create_connection<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return _ConnectionRecord(self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461, in __init__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.__connect(first_connect_check=True)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651, in __connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi connection = pool._invoke_creator(self)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 105, in connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return dialect.connect(*cargs, **cparams)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 393, in connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return self.dbapi.connect(*cargs, **cparams)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi return Connection(*args, **kwargs)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in __init__<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self.connect()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in connect<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi self._request_authentication()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in _request_authentication<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi auth_packet = self._read_packet()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi packet.check_error()<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi raise errorclass(errno, errval)<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')<br>2018-12-29 00:58:52.928 3765 ERROR nova.api.openstack.wsgi<br>2018-12-29 00:58:52.930 3765 INFO nova.api.openstack.wsgi [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] HTTP exception thrown: Unexpected API Error. Please report this at <a href="http://bugs.launchpad.net/nova/" target="_blank">http://bugs.launchpad.net/nova/</a> and attach the Nova API log if possible.<br><class 'sqlalchemy.exc.OperationalError'><br>2018-12-29 00:58:52.931 3765 INFO nova.osapi_compute.wsgi.server [req-666d4e30-081a-429f-88ae-763e62990770 bb1e571e4d64462bac80654b153a88c3 96ad10a59d114042b8f1ee82c438649a - default default] 10.150.21.195 "GET /v2.1/flavors/detail HTTP/1.1" status: 500 len: 638 time: 0.1569352</blockquote></div><div><br></div><div>I already did googling with 'Too many connections' but, there were only results about 'max_connections' of the mysql config file.</div><div>I also set the value as 4096 and I think it is enough.</div><div><br></div><div>Now, I do not know what to look for to resolve this problem.</div><div>Please give me any clue.</div><div><br></div><div>Thanks!</div><div>Regards,</div></div></div></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote><div> </div></div></div></div></div></div></div></div></div></div></div>