<div dir="ltr"><div><div>Hi, when using Mysql behind Haproxy, i have a problem on reboot when some nova services start after Haproxy service, but before Mysql service.<br></div><div>These service failed: (i re-checked for sure in /var/log/boot.log)<br>
* Starting Nova cert [fail]<br> * Starting Nova conductor [fail]<br> * Starting Nova scheduler [fail]<br>
* Starting Cinder scheduler server [fail]<br><br></div><div>I must login to server and re-start these services manually. <br></div><br></div>When check log of Nova-cert, I saw:<br><br>
<b>2013-07-12 15:20:05.020 2490 CRITICAL nova [-] (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communic<br>ation packet', system error: 0") None None</b><br>2013-07-12 15:20:05.020 2490 TRACE nova Traceback (most recent call last):<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/bin/nova-cert", line 51, in <module><br>2013-07-12 15:20:05.020 2490 TRACE nova service.wait()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 689, in wait<br>
2013-07-12 15:20:05.020 2490 TRACE nova _launcher.wait()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 209, in wait<br>2013-07-12 15:20:05.020 2490 TRACE nova super(ServiceLauncher, self).wait()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 179, in wait<br>2013-07-12 15:20:05.020 2490 TRACE nova service.wait()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait<br>
2013-07-12 15:20:05.020 2490 TRACE nova return self._exit_event.wait()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait<br>2013-07-12 15:20:05.020 2490 TRACE nova return hubs.get_hub().switch()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch<br>2013-07-12 15:20:05.020 2490 TRACE nova return self.greenlet.switch()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main<br>
2013-07-12 15:20:05.020 2490 TRACE nova result = function(*args, **kwargs)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 147, in run_server<br>2013-07-12 15:20:05.020 2490 TRACE nova server.start()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 434, in start<br>2013-07-12 15:20:05.020 2490 TRACE nova self.host, self.binary)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 261, in service_get_by_a<br>
rgs<br>2013-07-12 15:20:05.020 2490 TRACE nova binary=binary)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1348, in wrapper<br>2013-07-12 15:20:05.020 2490 TRACE nova return func(*args, **kwargs)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 424, in in<br>ner<br>2013-07-12 15:20:05.020 2490 TRACE nova return catch_client_exception(exceptions, func, *args, **kwargs)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 407, in ca<br>tch_client_exception<br>2013-07-12 15:20:05.020 2490 TRACE nova return func(*args, **kwargs)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 325, in service_get_<br>all_by<br>2013-07-12 15:20:05.020 2490 TRACE nova result = self.db.service_get_by_args(context, host, binary)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 155, in service_get_by_args<br>2013-07-12 15:20:05.020 2490 TRACE nova return IMPL.service_get_by_args(context, host, binary)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 96, in wrapper<br>2013-07-12 15:20:05.020 2490 TRACE nova return f(*args, **kwargs)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 409, in service_get_by_args<br>
2013-07-12 15:20:05.020 2490 TRACE nova result = model_query(context, models.Service).\<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 177, in model_query<br>
2013-07-12 15:20:05.020 2490 TRACE nova session = kwargs.get('session') or get_session()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 325, in get_session<br>
2013-07-12 15:20:05.020 2490 TRACE nova engine = get_engine()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 446, in get_engine<br>
2013-07-12 15:20:05.020 2490 TRACE nova _ENGINE = create_engine(CONF.sql_connection)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 562, in create_engine<br>
2013-07-12 15:20:05.020 2490 TRACE nova engine.connect()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2471, in connect<br>2013-07-12 15:20:05.020 2490 TRACE nova return self._connection_cls(self, **kwargs)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 878, in __init__<br>2013-07-12 15:20:05.020 2490 TRACE nova self.__connection = connection or engine.raw_connection()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2557, in raw_connection<br>2013-07-12 15:20:05.020 2490 TRACE nova return self.pool.unique_connection()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 184, in unique_connection<br>2013-07-12 15:20:05.020 2490 TRACE nova return _ConnectionFairy(self).checkout()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 401, in __init__<br>2013-07-12 15:20:05.020 2490 TRACE nova rec = self._connection_record = pool._do_get()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 746, in _do_get<br>2013-07-12 15:20:05.020 2490 TRACE nova con = self._create_connection()<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 189, in _create_connection<br>
2013-07-12 15:20:05.020 2490 TRACE nova return _ConnectionRecord(self)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 282, in __init__<br>2013-07-12 15:20:05.020 2490 TRACE nova self.connection = self.__connect()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 344, in __connect<br>2013-07-12 15:20:05.020 2490 TRACE nova connection = self.__pool._creator()<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect<br>2013-07-12 15:20:05.020 2490 TRACE nova return dialect.connect(*cargs, **cparams)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 281, in connect<br>2013-07-12 15:20:05.020 2490 TRACE nova return self.dbapi.connect(*cargs, **cparams)<br>
2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect<br>2013-07-12 15:20:05.020 2490 TRACE nova return Connection(*args, **kwargs)<br>2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__<br>
2013-07-12 15:20:05.020 2490 TRACE nova super(Connection, self).__init__(*args, **kwargs2)<br><b>2013-07-12 15:20:05.020 2490 TRACE nova OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None<br>
</b><div><br></div><div>Do you know a quick fix for this problem?<br></div><div>(I also send this email to report the problem)<br><br></div><div>Thanks you!<br><br></div><div>PS: I'm runing mysql, haproxy, nova-* services, cinder-* services on the same server, using Ubuntu 12.04.<br>
<br><br></div></div>