<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I tried 1 minute ago to create another instance.</div><div>Nova api reports the following:</div><div><br></div><div>RROR oslo_db.sqlalchemy.engines [req-cac96dee-d91b-48cb-831b-31f95cffa2f4 89f76bc5de5545f381da2c10c7df7f15 59f1f232ce28409593d66d8f6495e434 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: <a href="http://sqlalche.me/e/e3q8">http://sqlalche.me/e/e3q8</a>)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines context)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines context)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines result = self._query(query)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines conn.query(q)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines result.read()<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: <a href="http://sqlalche.me/e/e3q8">http://sqlalche.me/e/e3q8</a>)<br>2018-11-12 23:07:28.813 4224 ERROR oslo_db.sqlalchemy.engines <br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines [req-cac96dee-d91b-48cb-831b-31f95cffa2f4 89f76bc5de5545f381da2c10c7df7f15 59f1f232ce28409593d66d8f6495e434 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: <a href="http://sqlalche.me/e/e3q8">http://sqlalche.me/e/e3q8</a>)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1]))<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar()<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines context)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines context)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines result = self._query(query)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines conn.query(q)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines result.read()<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")<br>2018-11-12 23:07:28.831 4224 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: <a href="http://sqlalche.me/e/e3q8">http://sqlalche.me/e/e3q8</a>)</div><div><br></div><div><br></div><div>Anything went wrong while upgrade to queens in nova database ?</div><div><br></div><div># nova-manage db online_data_migrations</div><div><br></div><div>/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported<br> exception.NotSupportedWarning<br>Running batches of 50 until complete<br>1 rows matched query migrate_instances_add_request_spec, 0 migrated<br>+---------------------------------------------+--------------+-----------+<br>| Migration | Total Needed | Completed |<br>+---------------------------------------------+--------------+-----------+<br>| delete_build_requests_with_no_instance_uuid | 0 | 0 |<br>| migrate_aggregate_reset_autoincrement | 0 | 0 |<br>| migrate_aggregates | 0 | 0 |<br>| migrate_instance_groups_to_api_db | 0 | 0 |<br>| migrate_instances_add_request_spec | 1 | 0 |<br>| migrate_keypairs_to_api_db | 0 | 0 |<br>| migrate_quota_classes_to_api_db | 0 | 0 |<br>| migrate_quota_limits_to_api_db | 0 | 0 |<br>| migration_migrate_to_uuid | 0 | 0 |<br>| populate_missing_availability_zones | 0 | 0 |<br>| populate_uuids | 0 | 0 |<br>| service_uuids_online_data_migration | 0 | 0 |<br>+---------------------------------------------+--------------+-----------+<br><br></div><div><br></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Il giorno lun 12 nov 2018 alle ore 22:40 Slawomir Kaplonski <<a href="mailto:skaplons@redhat.com">skaplons@redhat.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
>From logs which You attached it looks that Your neutron-metadata-agent can’t connect to nova-api service. Please check if nova-metadata-api is reachable from node where Your neutron-metadata-agent is running.<br>
<br>
> Wiadomość napisana przez Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" target="_blank">ignaziocassano@gmail.com</a>> w dniu 12.11.2018, o godz. 22:34:<br>
> <br>
> Hello again,<br>
> I have another installation of ocata .<br>
> On ocata the metadata for a network id is displayed by ps -afe like this:<br>
> /usr/bin/python2 /bin/neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/c4731392-9b91-4663-adb3-b10b5ebcc4f1.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --network_id=c4731392-9b91-4663-adb3-b10b5ebcc4f1 --state_path=/var/lib/neutron --metadata_port=80 --metadata_proxy_user=996 --metadata_proxy_group=993 --log-file=neutron-ns-metadata-proxy-c4731392-9b91-4663-adb3-b10b5ebcc4f1.log --log-dir=/var/log/neutron<br>
> <br>
> On queens like this:<br>
> haproxy -f /var/lib/neutron/ns-metadata-proxy/e8ba8c09-a7dc-4a22-876e-b8d4187a23fe.conf<br>
> <br>
> Is it the correct behaviour ?<br>
<br>
Yes, that is correct. It was changed some time ago, see <a href="https://bugs.launchpad.net/neutron/+bug/1524916" rel="noreferrer" target="_blank">https://bugs.launchpad.net/neutron/+bug/1524916</a><br>
<br>
> <br>
> Regards<br>
> Ignazio<br>
> <br>
> <br>
> <br>
> Il giorno lun 12 nov 2018 alle ore 21:37 Slawomir Kaplonski <<a href="mailto:skaplons@redhat.com" target="_blank">skaplons@redhat.com</a>> ha scritto:<br>
> Hi,<br>
> <br>
> Can You share logs from Your haproxy-metadata-proxy service which is running in qdhcp namespace? There should be some info about reason of those errors 500.<br>
> <br>
> > Wiadomość napisana przez Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" target="_blank">ignaziocassano@gmail.com</a>> w dniu 12.11.2018, o godz. 19:49:<br>
> > <br>
> > Hi All,<br>
> > I upgraded manually my centos 7 openstack ocata to pike.<br>
> > All worked fine.<br>
> > Then I upgraded from pike to Queens and instances stopped to reach metadata on 169.254.169.254 with error 500.<br>
> > I am using isolated metadata true in my dhcp conf and in dhcp namespace the port 80 is in listen.<br>
> > Please, anyone can help me?<br>
> > Regards<br>
> > Ignazio<br>
> > <br>
> > _______________________________________________<br>
> > OpenStack-operators mailing list<br>
> > <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
> <br>
> — <br>
> Slawek Kaplonski<br>
> Senior software engineer<br>
> Red Hat<br>
> <br>
<br>
— <br>
Slawek Kaplonski<br>
Senior software engineer<br>
Red Hat<br>
<br>
</blockquote></div>