2025-03-13 11:21:18.157 43002 INFO neutron.common.config [-] Logging enabled! 2025-03-13 11:21:18.157 43002 INFO neutron.common.config [-] /usr/bin/neutron-l3-agent version 20.5.0 2025-03-13 11:21:18.484 43002 INFO neutron.agent.l3.agent [req-ed7cf7d7-bdf5-46f6-9381-0bc3c02b302d - - - - -] Agent HA routers count 0 2025-03-13 11:21:18.489 43002 INFO neutron.agent.agent_extensions_manager [req-ed7cf7d7-bdf5-46f6-9381-0bc3c02b302d - - - - -] Loaded agent extensions: ['fip_qos'] 2025-03-13 11:21:18.489 43002 INFO neutron.agent.agent_extensions_manager [req-ed7cf7d7-bdf5-46f6-9381-0bc3c02b302d - - - - -] Initializing agent extension 'fip_qos' 2025-03-13 11:21:18.524 43002 WARNING oslo_config.cfg [-] Deprecated: Option "allow_overlapping_ips" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. 2025-03-13 11:21:18.530 43002 INFO eventlet.wsgi.server [-] (43002) wsgi starting up on http:/var/lib/neutron/keepalived-state-change 2025-03-13 11:21:18.587 43002 INFO neutron.agent.l3.agent [-] L3 agent started 2025-03-13 11:21:22.591 43002 INFO oslo.privsep.daemon [req-4a643535-a310-44d4-9b89-dcc5f7bdc49e - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.namespace_cmd', '--privsep_sock_path', '/tmp/tmpo86bruss/privsep.sock'] 2025-03-13 11:21:23.200 43002 INFO oslo.privsep.daemon [req-4a643535-a310-44d4-9b89-dcc5f7bdc49e - - - - -] Spawned new privsep daemon via rootwrap 2025-03-13 11:21:23.105 43092 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-13 11:21:23.108 43092 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-13 11:21:23.109 43092 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2025-03-13 11:21:23.110 43092 INFO oslo.privsep.daemon [-] privsep daemon running as pid 43092 2025-03-13 12:28:17.547 43002 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 56723ddd-90fe-479c-887e-2719a868a618. Wait time elapsed: 0.000 2025-03-13 12:28:17.547 43002 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 56723ddd-90fe-479c-887e-2719a868a618. Wait time elapsed: 0.000 2025-03-13 12:28:17.548 43002 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 41f41dde-923d-4fe7-a5b0-3243fbf7a966. Wait time elapsed: 0.001 2025-03-13 12:28:17.548 43002 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 41f41dde-923d-4fe7-a5b0-3243fbf7a966. Wait time elapsed: 0.001 2025-03-13 12:28:17.549 43002 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 81563e05-93e2-432b-921f-7dfe313621b6. Wait time elapsed: 0.002 2025-03-13 12:28:17.549 43002 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 81563e05-93e2-432b-921f-7dfe313621b6. Wait time elapsed: 0.002 2025-03-13 12:28:17.550 43002 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 7d4b1754-21f0-4095-a450-1e101c01bf84. Wait time elapsed: 0.003 2025-03-13 12:28:17.550 43002 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 7d4b1754-21f0-4095-a450-1e101c01bf84. Wait time elapsed: 0.003 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent raise result 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-13 12:28:18.041 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:18.042 43002 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id b549b0a6-f788-4aeb-862a-0a62513ae452. Wait time elapsed: 0.371 2025-03-13 12:28:18.042 43002 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id b549b0a6-f788-4aeb-862a-0a62513ae452. Wait time elapsed: 0.371 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent raise result 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-13 12:28:18.059 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:18.060 43002 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 5a22b88c-8344-458a-84eb-4e6975e4c753. Wait time elapsed: 0.389 2025-03-13 12:28:18.060 43002 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 5a22b88c-8344-458a-84eb-4e6975e4c753. Wait time elapsed: 0.389 2025-03-13 12:28:18.563 43002 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp0g4o2kak/privsep.sock'] 2025-03-13 12:28:19.202 43002 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-13 12:28:19.082 79921 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-13 12:28:19.085 79921 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-13 12:28:19.086 79921 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2025-03-13 12:28:19.087 79921 INFO oslo.privsep.daemon [-] privsep daemon running as pid 79921 2025-03-13 12:28:19.204 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:19.205 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:19.205 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:19.829 43002 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.link_cmd', '--privsep_sock_path', '/tmp/tmp5f9e7_1d/privsep.sock'] 2025-03-13 12:28:19.925 43002 WARNING neutron.agent.l3.dvr_local_router [req-6320986f-a9fb-4f93-b9a2-c1c781083760 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-5be154bc-e7 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-13 12:28:19.926 43002 WARNING neutron.agent.l3.dvr_local_router [req-6320986f-a9fb-4f93-b9a2-c1c781083760 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-8e2c27c0-75 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-13 12:28:20.465 43002 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-13 12:28:20.348 80012 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-13 12:28:20.350 80012 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-13 12:28:20.352 80012 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2025-03-13 12:28:20.352 80012 INFO oslo.privsep.daemon [-] privsep daemon running as pid 80012 2025-03-13 12:28:20.468 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:20.468 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:20.468 43002 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info [-] Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info self.process_external() 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 742, in process_external 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info self.connect_rtr_2_fip() 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 757, in connect_rtr_2_fip 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self) 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 434, in create_rtr_2_fip_link 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info rtr_2_fip_dev.link.set_mtu(mtu) 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 454, in set_mtu 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info privileged.set_link_attribute( 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info raise exc_type(*result[2]) 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.632 43002 ERROR neutron.agent.l3.router_info 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent [-] Error while processing router e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 646, in _process_added_router 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent ri.process() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 927, in process 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_router_base.py", line 29, in process 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 180, in call 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent self.logger(e) 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent self.process_external() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 742, in process_external 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent self.connect_rtr_2_fip() 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 757, in connect_rtr_2_fip 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent self.fip_ns.create_rtr_2_fip_link(self) 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 434, in create_rtr_2_fip_link 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent rtr_2_fip_dev.link.set_mtu(mtu) 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 454, in set_mtu 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent privileged.set_link_attribute( 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent raise exc_type(*result[2]) 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.634 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns [-] DVR: Gateway update in FIP namespace failed: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns Traceback (most recent call last): 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 144, in create_or_update_gateway_port 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns self._update_gateway_port( 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 315, in _update_gateway_port 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns self.driver.set_onlink_routes( 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 194, in set_onlink_routes 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns onlink = device.route.list_onlink_routes(constants.IP_VERSION_4) 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 633, in list_onlink_routes 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns routes = self.list_routes(ip_version, scope='link') 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 629, in list_routes 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns return list_ip_routes(self._parent.namespace, ip_version, scope=scope, 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 1550, in list_ip_routes 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns routes = privileged.list_ip_routes(namespace, ip_version, device=device, 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns return self(f, *args, **kw) 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns do = self.iter(retry_state=retry_state) 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns return fut.result() 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns return self.__get_result() 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns raise self._exception 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns result = fn(*args, **kwargs) 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns raise exc_type(*result[2]) 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.698 43002 ERROR neutron.agent.l3.dvr_fip_ns 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.dvr_fip_ns [-] DVR: FIP namespace config failure for interface fg-e833056a-a6: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info [-] Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self.process_external() 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 741, in process_external 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self.create_dvr_external_gateway_on_agent(ex_gw_port) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 843, in create_dvr_external_gateway_on_agent 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self.fip_ns.create_or_update_gateway_port(fip_agent_port) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 153, in create_or_update_gateway_port 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info LOG.exception('DVR: Gateway update in ' 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self.force_reraise() 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info raise self.value 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 144, in create_or_update_gateway_port 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self._update_gateway_port( 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 315, in _update_gateway_port 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info self.driver.set_onlink_routes( 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 194, in set_onlink_routes 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info onlink = device.route.list_onlink_routes(constants.IP_VERSION_4) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 633, in list_onlink_routes 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info routes = self.list_routes(ip_version, scope='link') 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 629, in list_routes 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return list_ip_routes(self._parent.namespace, ip_version, scope=scope, 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 1550, in list_ip_routes 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info routes = privileged.list_ip_routes(namespace, ip_version, device=device, 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return self(f, *args, **kw) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info do = self.iter(retry_state=retry_state) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return fut.result() 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return self.__get_result() 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info raise self._exception 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info result = fn(*args, **kwargs) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info raise exc_type(*result[2]) 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.700 43002 ERROR neutron.agent.l3.router_info 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent [-] Error while processing router ebd75b85-e4e9-4c99-9966-17a1ea1a3b14: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 646, in _process_added_router 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent ri.process() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 927, in process 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_router_base.py", line 29, in process 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 180, in call 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.logger(e) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.process_external() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 741, in process_external 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.create_dvr_external_gateway_on_agent(ex_gw_port) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 843, in create_dvr_external_gateway_on_agent 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.fip_ns.create_or_update_gateway_port(fip_agent_port) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 153, in create_or_update_gateway_port 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent LOG.exception('DVR: Gateway update in ' 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 144, in create_or_update_gateway_port 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self._update_gateway_port( 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 315, in _update_gateway_port 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent self.driver.set_onlink_routes( 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 194, in set_onlink_routes 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent onlink = device.route.list_onlink_routes(constants.IP_VERSION_4) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 633, in list_onlink_routes 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent routes = self.list_routes(ip_version, scope='link') 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 629, in list_routes 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return list_ip_routes(self._parent.namespace, ip_version, scope=scope, 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 1550, in list_ip_routes 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent routes = privileged.list_ip_routes(namespace, ip_version, device=device, 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return self(f, *args, **kw) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent do = self.iter(retry_state=retry_state) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return fut.result() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return self.__get_result() 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent raise self._exception 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent result = fn(*args, **kwargs) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent raise exc_type(*result[2]) 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.701 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: ebd75b85-e4e9-4c99-9966-17a1ea1a3b14: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self._process_added_router(router) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 656, in _process_added_router 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self._cleanup_failed_router(router['id'], 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 646, in _process_added_router 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent ri.process() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 927, in process 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_router_base.py", line 29, in process 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 180, in call 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.logger(e) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.process_external() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 741, in process_external 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.create_dvr_external_gateway_on_agent(ex_gw_port) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 843, in create_dvr_external_gateway_on_agent 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.fip_ns.create_or_update_gateway_port(fip_agent_port) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 153, in create_or_update_gateway_port 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent LOG.exception('DVR: Gateway update in ' 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 144, in create_or_update_gateway_port 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self._update_gateway_port( 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 315, in _update_gateway_port 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent self.driver.set_onlink_routes( 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 194, in set_onlink_routes 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent onlink = device.route.list_onlink_routes(constants.IP_VERSION_4) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 633, in list_onlink_routes 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent routes = self.list_routes(ip_version, scope='link') 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 629, in list_routes 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return list_ip_routes(self._parent.namespace, ip_version, scope=scope, 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 1550, in list_ip_routes 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent routes = privileged.list_ip_routes(namespace, ip_version, device=device, 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return self(f, *args, **kw) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent do = self.iter(retry_state=retry_state) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return fut.result() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return self.__get_result() 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent raise self._exception 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent result = fn(*args, **kwargs) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent raise exc_type(*result[2]) 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fg-e833056a-a6 not found in namespace fip-25ce6d25-9b06-4c7c-8512-38b823436183. 2025-03-13 12:28:24.781 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:24.782 43002 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 71c6df8b-b175-4e1a-998e-84cd15ad9387. Wait time elapsed: 7.110 2025-03-13 12:28:24.782 43002 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 71c6df8b-b175-4e1a-998e-84cd15ad9387. Wait time elapsed: 7.111 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self._process_added_router(router) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 656, in _process_added_router 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self._cleanup_failed_router(router['id'], 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 646, in _process_added_router 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent ri.process() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 927, in process 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_router_base.py", line 29, in process 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 180, in call 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.logger(e) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.force_reraise() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent raise self.value 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1304, in process 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.process_external() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 742, in process_external 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.connect_rtr_2_fip() 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_local_router.py", line 757, in connect_rtr_2_fip 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent self.fip_ns.create_rtr_2_fip_link(self) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 434, in create_rtr_2_fip_link 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent rtr_2_fip_dev.link.set_mtu(mtu) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 454, in set_mtu 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent privileged.set_link_attribute( 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs, 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent raise exc_type(*result[2]) 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface rfp-e3fb6ba3-7 not found in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3. 2025-03-13 12:28:24.825 43002 ERROR neutron.agent.l3.agent 2025-03-13 12:28:24.825 43002 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id ec4ef363-6586-4a00-9018-86815df366f4. Wait time elapsed: 6.384 2025-03-13 12:28:24.826 43002 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id ec4ef363-6586-4a00-9018-86815df366f4. Wait time elapsed: 6.384 2025-03-13 12:28:24.936 43002 WARNING neutron.agent.linux.iptables_manager [-] Duplicate iptables rule detected. This may indicate a bug in the iptables rule generation code. Line: -A neutron-l3-agent-PREROUTING -j CT --notrack 2025-03-13 12:28:25.841 43002 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'arping', '-U', '-I', 'qr-cd332bdb-a2', '-c', 1, '-w', 2, '172.19.3.33']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3": No such file or directory 2025-03-13 12:28:26.098 43002 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 172.19.3.33 on qr-cd332bdb-a2 in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3: Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'arping', '-U', '-I', 'qr-cd332bdb-a2', '-c', 1, '-w', 2, '172.19.3.33']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3": No such file or directory 2025-03-13 12:28:26.098 43002 INFO neutron.agent.linux.ip_lib [-] Interface qr-cd332bdb-a2 or address 172.19.3.33 in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3 was deleted concurrently 2025-03-13 12:28:26.614 43002 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'arping', '-U', '-I', 'qr-f269cf3d-8a', '-c', 1, '-w', 2, '172.31.3.31']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3": No such file or directory 2025-03-13 12:28:26.630 43002 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 172.31.3.31 on qr-f269cf3d-8a in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3: Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'arping', '-U', '-I', 'qr-f269cf3d-8a', '-c', 1, '-w', 2, '172.31.3.31']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3": No such file or directory 2025-03-13 12:28:26.630 43002 INFO neutron.agent.linux.ip_lib [-] Interface qr-f269cf3d-8a or address 172.31.3.31 in namespace qrouter-e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3 was deleted concurrently 2025-03-13 12:28:26.928 43002 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=81316 2025-03-13 12:28:26.974 43002 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 81563e05-93e2-432b-921f-7dfe313621b6. Time elapsed: 9.426 2025-03-13 12:28:26.975 43002 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id b3568968-8758-4240-9e87-7d836299ec9f. Wait time elapsed: 8.413 2025-03-13 12:28:26.975 43002 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id b3568968-8758-4240-9e87-7d836299ec9f. Wait time elapsed: 8.413 2025-03-13 12:28:26.987 43002 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id b549b0a6-f788-4aeb-862a-0a62513ae452. Time elapsed: 8.945 2025-03-13 12:28:26.987 43002 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id e65a809b-beeb-46df-b417-88976be2af75. Wait time elapsed: 7.858 2025-03-13 12:28:26.987 43002 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id e65a809b-beeb-46df-b417-88976be2af75. Wait time elapsed: 7.859 2025-03-13 12:28:27.725 43002 INFO neutron.agent.linux.interface [-] Device fg-0a5d1e67-87 already exists 2025-03-13 12:28:28.446 43002 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id e65a809b-beeb-46df-b417-88976be2af75. Time elapsed: 1.458 2025-03-13 12:28:28.490 43002 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id b3568968-8758-4240-9e87-7d836299ec9f. Time elapsed: 1.515 2025-03-13 12:28:28.490 43002 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id b6b0c7e3-ed89-42d1-b59c-ce0742da4b78. Wait time elapsed: 1.504 2025-03-13 12:28:28.491 43002 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id b6b0c7e3-ed89-42d1-b59c-ce0742da4b78. Wait time elapsed: 1.504 2025-03-13 12:28:28.948 43002 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 71c6df8b-b175-4e1a-998e-84cd15ad9387. Time elapsed: 4.167 2025-03-13 12:28:28.949 43002 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 00452739-dd0e-43e5-b33a-731512a7ddaa. Wait time elapsed: 1.975 2025-03-13 12:28:28.949 43002 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 00452739-dd0e-43e5-b33a-731512a7ddaa. Wait time elapsed: 1.975 2025-03-13 12:28:29.344 43002 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id ec4ef363-6586-4a00-9018-86815df366f4. Time elapsed: 4.519 2025-03-13 12:28:29.345 43002 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 096e5935-b9ad-475a-b1c0-282cd5c8d5c4. Wait time elapsed: 2.370 2025-03-13 12:28:29.345 43002 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 096e5935-b9ad-475a-b1c0-282cd5c8d5c4. Wait time elapsed: 2.371 2025-03-13 12:28:29.836 43002 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 00452739-dd0e-43e5-b33a-731512a7ddaa. Time elapsed: 0.888 2025-03-13 12:28:30.532 43002 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 096e5935-b9ad-475a-b1c0-282cd5c8d5c4. Time elapsed: 1.188 2025-03-13 12:28:30.605 43002 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id b6b0c7e3-ed89-42d1-b59c-ce0742da4b78. Time elapsed: 2.115 2025-03-13 13:52:41.878 43002 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 115d861a-a9cf-49da-965e-6e31d7371c87. Wait time elapsed: 0.001 2025-03-13 13:52:41.879 43002 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 115d861a-a9cf-49da-965e-6e31d7371c87. Wait time elapsed: 0.002 2025-03-13 13:52:41.881 43002 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id f1b32e47-897f-493b-a864-aa39e25cd405. Wait time elapsed: 0.001 2025-03-13 13:52:41.881 43002 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id f1b32e47-897f-493b-a864-aa39e25cd405. Wait time elapsed: 0.001 2025-03-13 13:52:41.882 43002 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 3c9e4e2e-1282-4b88-87fe-45c1a8b82196. Wait time elapsed: 0.000 2025-03-13 13:52:41.882 43002 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 3c9e4e2e-1282-4b88-87fe-45c1a8b82196. Wait time elapsed: 0.000 2025-03-13 13:52:41.883 43002 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 626da032-a37d-4961-9967-40c9d12fdda1. Wait time elapsed: 0.001 2025-03-13 13:52:41.883 43002 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 626da032-a37d-4961-9967-40c9d12fdda1. Wait time elapsed: 0.001 2025-03-13 13:52:42.805 43002 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id f1b32e47-897f-493b-a864-aa39e25cd405. Time elapsed: 0.924 2025-03-13 13:52:42.831 43002 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 115d861a-a9cf-49da-965e-6e31d7371c87. Time elapsed: 0.953 2025-03-13 13:52:42.920 43002 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 3c9e4e2e-1282-4b88-87fe-45c1a8b82196. Time elapsed: 1.038 2025-03-13 13:52:42.946 43002 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 626da032-a37d-4961-9967-40c9d12fdda1. Time elapsed: 1.063 2025-03-13 16:04:55.864 43002 INFO oslo_rootwrap.client [-] Rootwrap daemon process exit with status: -1 2025-03-13 16:04:59.139 43002 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored 2025-03-13 16:04:59.140 43002 INFO neutron.agent.l3.agent [-] Stopping L3 agent 2025-03-13 16:09:42.844 2013 INFO neutron.common.config [-] Logging enabled! 2025-03-13 16:09:42.847 2013 INFO neutron.common.config [-] /usr/bin/neutron-l3-agent version 20.5.0 2025-03-13 16:09:46.193 2013 ERROR oslo.messaging._drivers.impl_rabbit [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH 2025-03-13 16:09:46.257 2013 ERROR oslo.messaging._drivers.impl_rabbit [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 0 seconds): OSError: [Errno 113] EHOSTUNREACH 2025-03-13 16:09:49.137 2013 ERROR oslo.messaging._drivers.impl_rabbit [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 1.0 seconds): OSError: [Errno 113] EHOSTUNREACH 2025-03-13 16:09:51.350 2013 INFO neutron.agent.l3.agent [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Agent HA routers count 0 2025-03-13 16:09:51.356 2013 INFO neutron.agent.agent_extensions_manager [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Loaded agent extensions: ['fip_qos'] 2025-03-13 16:09:51.356 2013 INFO neutron.agent.agent_extensions_manager [req-8f5adb13-fcb6-434a-94ea-b7102096a4eb - - - - -] Initializing agent extension 'fip_qos' 2025-03-13 16:09:51.392 2013 WARNING oslo_config.cfg [-] Deprecated: Option "allow_overlapping_ips" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. 2025-03-13 16:09:51.398 2013 INFO eventlet.wsgi.server [-] (2013) wsgi starting up on http:/var/lib/neutron/keepalived-state-change 2025-03-13 16:09:51.439 2013 INFO neutron.agent.l3.agent [-] Agent has just been revived. Doing a full sync. 2025-03-13 16:09:51.456 2013 INFO neutron.agent.l3.agent [-] L3 agent started 2025-03-13 16:09:52.459 2013 INFO oslo.privsep.daemon [req-30d16575-92af-4dd8-bf6b-cce02ecb0360 - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.namespace_cmd', '--privsep_sock_path', '/tmp/tmp00xjbpvk/privsep.sock'] 2025-03-13 16:09:53.076 2013 INFO oslo.privsep.daemon [req-30d16575-92af-4dd8-bf6b-cce02ecb0360 - - - - -] Spawned new privsep daemon via rootwrap 2025-03-13 16:09:52.981 3322 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-13 16:09:52.984 3322 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-13 16:09:52.985 3322 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2025-03-13 16:09:52.986 3322 INFO oslo.privsep.daemon [-] privsep daemon running as pid 3322 2025-03-14 08:06:26.143 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d6a7e3ec-7600-412d-8f83-99c562129b1a. Wait time elapsed: 0.000 2025-03-14 08:06:26.143 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d6a7e3ec-7600-412d-8f83-99c562129b1a. Wait time elapsed: 0.000 2025-03-14 08:06:26.144 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 7be2a779-4282-47d2-9b18-0ad0a38ca0fb. Wait time elapsed: 0.001 2025-03-14 08:06:26.144 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 7be2a779-4282-47d2-9b18-0ad0a38ca0fb. Wait time elapsed: 0.001 2025-03-14 08:06:26.145 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 0bb4629b-37e3-4f05-921f-a4101edf3473. Wait time elapsed: 0.002 2025-03-14 08:06:26.145 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 0bb4629b-37e3-4f05-921f-a4101edf3473. Wait time elapsed: 0.002 2025-03-14 08:06:26.146 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 710d6134-06f2-44bd-9267-5fb4811680e8. Wait time elapsed: 0.003 2025-03-14 08:06:26.146 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 710d6134-06f2-44bd-9267-5fb4811680e8. Wait time elapsed: 0.003 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:06:26.742 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:06:26.745 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 54c356c2-aac6-42c1-b0fe-45a5803fd8ab. Wait time elapsed: 0.475 2025-03-14 08:06:26.745 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 54c356c2-aac6-42c1-b0fe-45a5803fd8ab. Wait time elapsed: 0.475 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:06:26.877 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:06:26.877 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 04cd3aaa-f526-4560-8137-b95302834acc. Wait time elapsed: 0.607 2025-03-14 08:06:26.878 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 04cd3aaa-f526-4560-8137-b95302834acc. Wait time elapsed: 0.608 2025-03-14 08:06:27.225 2013 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpldtw0usy/privsep.sock'] 2025-03-14 08:06:27.823 2013 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-14 08:06:27.731 313514 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-14 08:06:27.733 313514 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-14 08:06:27.735 313514 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2025-03-14 08:06:27.735 313514 INFO oslo.privsep.daemon [-] privsep daemon running as pid 313514 2025-03-14 08:06:28.439 2013 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.link_cmd', '--privsep_sock_path', '/tmp/tmpd52iyyln/privsep.sock'] 2025-03-14 08:06:28.539 2013 WARNING neutron.agent.l3.dvr_local_router [req-cd62aff8-9a31-42d7-8fcf-ecbf6af4c1e3 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-5be154bc-e7 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-14 08:06:28.541 2013 WARNING neutron.agent.l3.dvr_local_router [req-cd62aff8-9a31-42d7-8fcf-ecbf6af4c1e3 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-8e2c27c0-75 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-14 08:06:29.070 2013 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-14 08:06:28.968 313552 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-14 08:06:28.971 313552 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-14 08:06:28.972 313552 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2025-03-14 08:06:28.973 313552 INFO oslo.privsep.daemon [-] privsep daemon running as pid 313552 2025-03-14 08:06:29.072 2013 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 08:06:29.073 2013 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 08:06:29.073 2013 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 08:06:34.470 2013 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=314578 2025-03-14 08:06:34.531 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d6a7e3ec-7600-412d-8f83-99c562129b1a. Time elapsed: 8.388 2025-03-14 08:06:34.531 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ddf6ab51-ecc8-4379-b404-ac5444f2461a. Wait time elapsed: 8.261 2025-03-14 08:06:34.531 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ddf6ab51-ecc8-4379-b404-ac5444f2461a. Wait time elapsed: 8.262 2025-03-14 08:06:34.540 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 54c356c2-aac6-42c1-b0fe-45a5803fd8ab. Time elapsed: 7.796 2025-03-14 08:06:34.541 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 51caac16-5b07-4837-a9b7-b1c40c230c9d. Wait time elapsed: 7.783 2025-03-14 08:06:34.541 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 51caac16-5b07-4837-a9b7-b1c40c230c9d. Wait time elapsed: 7.783 2025-03-14 08:06:34.956 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 0bb4629b-37e3-4f05-921f-a4101edf3473. Time elapsed: 8.811 2025-03-14 08:06:34.956 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 15a378d1-b56e-4bec-8f3c-e30bb68115c2. Wait time elapsed: 7.732 2025-03-14 08:06:34.956 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 15a378d1-b56e-4bec-8f3c-e30bb68115c2. Wait time elapsed: 7.733 2025-03-14 08:06:35.228 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 04cd3aaa-f526-4560-8137-b95302834acc. Time elapsed: 8.351 2025-03-14 08:06:35.228 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 70081cfb-f864-41ba-926b-a496325d7796. Wait time elapsed: 7.396 2025-03-14 08:06:35.229 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 70081cfb-f864-41ba-926b-a496325d7796. Wait time elapsed: 7.396 2025-03-14 08:06:35.483 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id ddf6ab51-ecc8-4379-b404-ac5444f2461a. Time elapsed: 0.952 2025-03-14 08:06:35.483 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 6547e19a-df20-4482-9596-81f6aad917c4. Wait time elapsed: 0.528 2025-03-14 08:06:35.483 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 6547e19a-df20-4482-9596-81f6aad917c4. Wait time elapsed: 0.528 2025-03-14 08:06:35.754 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 51caac16-5b07-4837-a9b7-b1c40c230c9d. Time elapsed: 1.213 2025-03-14 08:06:35.754 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id d12a2d03-f285-4b48-8c36-af480c874df1. Wait time elapsed: 0.799 2025-03-14 08:06:35.755 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id d12a2d03-f285-4b48-8c36-af480c874df1. Wait time elapsed: 0.799 2025-03-14 08:06:36.301 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 15a378d1-b56e-4bec-8f3c-e30bb68115c2. Time elapsed: 1.345 2025-03-14 08:06:36.302 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id efad88a7-635a-4ccb-9793-f5f481ed1bdc. Wait time elapsed: 1.079 2025-03-14 08:06:36.302 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id efad88a7-635a-4ccb-9793-f5f481ed1bdc. Wait time elapsed: 1.080 2025-03-14 08:06:36.426 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 70081cfb-f864-41ba-926b-a496325d7796. Time elapsed: 1.197 2025-03-14 08:06:36.426 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id a0936c11-8724-4f5a-9b60-e1ad856c5d16. Wait time elapsed: 0.927 2025-03-14 08:06:36.426 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id a0936c11-8724-4f5a-9b60-e1ad856c5d16. Wait time elapsed: 0.927 2025-03-14 08:06:36.794 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 6547e19a-df20-4482-9596-81f6aad917c4. Time elapsed: 1.311 2025-03-14 08:06:36.930 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id d12a2d03-f285-4b48-8c36-af480c874df1. Time elapsed: 1.175 2025-03-14 08:06:37.623 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id a0936c11-8724-4f5a-9b60-e1ad856c5d16. Time elapsed: 1.197 2025-03-14 08:06:37.767 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id efad88a7-635a-4ccb-9793-f5f481ed1bdc. Time elapsed: 1.465 2025-03-14 08:13:42.412 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 8fa41ca8-7d39-46c3-b689-0b8e37c6ac54. Wait time elapsed: 0.001 2025-03-14 08:13:42.412 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 8fa41ca8-7d39-46c3-b689-0b8e37c6ac54. Wait time elapsed: 0.001 2025-03-14 08:13:42.413 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 02724691-c62c-4b98-87e5-79f60d534223. Wait time elapsed: 0.002 2025-03-14 08:13:42.413 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 02724691-c62c-4b98-87e5-79f60d534223. Wait time elapsed: 0.002 2025-03-14 08:13:42.414 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 78ed133d-d7db-482f-a8f0-169678c13ebe. Wait time elapsed: 0.001 2025-03-14 08:13:42.415 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 78ed133d-d7db-482f-a8f0-169678c13ebe. Wait time elapsed: 0.001 2025-03-14 08:13:42.415 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id ff6ebe20-4eef-47c4-8481-798896e3f385. Wait time elapsed: 0.000 2025-03-14 08:13:42.415 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id ff6ebe20-4eef-47c4-8481-798896e3f385. Wait time elapsed: 0.000 2025-03-14 08:13:43.344 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 02724691-c62c-4b98-87e5-79f60d534223. Time elapsed: 0.931 2025-03-14 08:13:43.426 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 8fa41ca8-7d39-46c3-b689-0b8e37c6ac54. Time elapsed: 1.015 2025-03-14 08:13:43.435 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 78ed133d-d7db-482f-a8f0-169678c13ebe. Time elapsed: 1.021 2025-03-14 08:13:43.518 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id ff6ebe20-4eef-47c4-8481-798896e3f385. Time elapsed: 1.103 2025-03-14 08:15:00.572 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id ec784cac-5676-404f-ba18-a5c37378cd4a. Wait time elapsed: 0.000 2025-03-14 08:15:00.572 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id ec784cac-5676-404f-ba18-a5c37378cd4a. Wait time elapsed: 0.000 2025-03-14 08:15:00.573 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 610e7e0d-a206-4d4f-b43d-3eb6cd7a81aa. Wait time elapsed: 0.001 2025-03-14 08:15:00.573 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 610e7e0d-a206-4d4f-b43d-3eb6cd7a81aa. Wait time elapsed: 0.001 2025-03-14 08:15:00.574 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 06d60462-24e1-459f-b900-f2fda0a14679. Wait time elapsed: 0.002 2025-03-14 08:15:00.574 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 06d60462-24e1-459f-b900-f2fda0a14679. Wait time elapsed: 0.002 2025-03-14 08:15:00.575 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id a3f80e9f-50dd-4b91-abcd-24339fcdee7f. Wait time elapsed: 0.003 2025-03-14 08:15:00.575 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id a3f80e9f-50dd-4b91-abcd-24339fcdee7f. Wait time elapsed: 0.003 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:15:01.143 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:15:01.144 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 8c47f3a1-96dd-4359-9f1e-d7036b0cda68. Wait time elapsed: 0.446 2025-03-14 08:15:01.144 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 8c47f3a1-96dd-4359-9f1e-d7036b0cda68. Wait time elapsed: 0.446 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:15:01.145 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:15:01.145 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id f4c85b06-9945-442e-9af6-b501be9ee30b. Wait time elapsed: 0.447 2025-03-14 08:15:01.146 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id f4c85b06-9945-442e-9af6-b501be9ee30b. Wait time elapsed: 0.447 2025-03-14 08:15:05.409 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id a3f80e9f-50dd-4b91-abcd-24339fcdee7f. Time elapsed: 4.834 2025-03-14 08:15:05.409 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 3af5044c-3b1f-40dc-b952-d3130729d6f1. Wait time elapsed: 4.711 2025-03-14 08:15:05.409 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 3af5044c-3b1f-40dc-b952-d3130729d6f1. Wait time elapsed: 4.711 2025-03-14 08:15:06.250 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id f4c85b06-9945-442e-9af6-b501be9ee30b. Time elapsed: 5.104 2025-03-14 08:15:06.250 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id c7e9178b-5faa-4f06-8eb3-ed9ca06fbbb5. Wait time elapsed: 3.727 2025-03-14 08:15:06.251 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id c7e9178b-5faa-4f06-8eb3-ed9ca06fbbb5. Wait time elapsed: 3.727 2025-03-14 08:15:06.482 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 3af5044c-3b1f-40dc-b952-d3130729d6f1. Time elapsed: 1.073 2025-03-14 08:15:07.335 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 8c47f3a1-96dd-4359-9f1e-d7036b0cda68. Time elapsed: 6.191 2025-03-14 08:15:07.336 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 5d2a73b5-2e8d-46d3-aa76-e4f55102006d. Wait time elapsed: 5.678 2025-03-14 08:15:07.336 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 5d2a73b5-2e8d-46d3-aa76-e4f55102006d. Wait time elapsed: 5.678 2025-03-14 08:15:07.337 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id d89ec844-8210-498c-bf68-688ac80c5c71. Wait time elapsed: 0.001 2025-03-14 08:15:07.337 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id d89ec844-8210-498c-bf68-688ac80c5c71. Wait time elapsed: 0.002 2025-03-14 08:15:07.481 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id c7e9178b-5faa-4f06-8eb3-ed9ca06fbbb5. Time elapsed: 1.231 2025-03-14 08:15:07.481 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id ec81749f-baaf-4060-bb57-9859d57bd149. Wait time elapsed: 0.146 2025-03-14 08:15:07.481 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id ec81749f-baaf-4060-bb57-9859d57bd149. Wait time elapsed: 0.146 2025-03-14 08:15:07.531 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id ec784cac-5676-404f-ba18-a5c37378cd4a. Time elapsed: 6.959 2025-03-14 08:15:07.531 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id d6e6531e-c5d5-491b-9097-b291332d3489. Wait time elapsed: 5.519 2025-03-14 08:15:07.532 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id d6e6531e-c5d5-491b-9097-b291332d3489. Wait time elapsed: 5.519 2025-03-14 08:15:08.384 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d89ec844-8210-498c-bf68-688ac80c5c71. Time elapsed: 1.048 2025-03-14 08:15:08.600 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 5d2a73b5-2e8d-46d3-aa76-e4f55102006d. Time elapsed: 1.265 2025-03-14 08:15:08.601 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e5323fe8-f55b-44fe-b7da-e98c12ab6ef7. Wait time elapsed: 1.070 2025-03-14 08:15:08.601 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e5323fe8-f55b-44fe-b7da-e98c12ab6ef7. Wait time elapsed: 1.070 2025-03-14 08:15:08.847 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id ec81749f-baaf-4060-bb57-9859d57bd149. Time elapsed: 1.366 2025-03-14 08:15:08.847 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 22f2b841-b600-443b-99e4-00d836dd69ea. Wait time elapsed: 1.316 2025-03-14 08:15:08.848 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 22f2b841-b600-443b-99e4-00d836dd69ea. Wait time elapsed: 1.317 2025-03-14 08:15:08.876 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id d6e6531e-c5d5-491b-9097-b291332d3489. Time elapsed: 1.344 2025-03-14 08:15:09.970 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id e5323fe8-f55b-44fe-b7da-e98c12ab6ef7. Time elapsed: 1.369 2025-03-14 08:15:10.096 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 22f2b841-b600-443b-99e4-00d836dd69ea. Time elapsed: 1.249 2025-03-14 08:18:23.157 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id dfccaa46-5671-4a30-acfb-b64b87580eff. Wait time elapsed: 0.000 2025-03-14 08:18:23.157 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id dfccaa46-5671-4a30-acfb-b64b87580eff. Wait time elapsed: 0.001 2025-03-14 08:18:23.158 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 8dbdfa8e-53d3-48ba-b0d4-a62eee529f3c. Wait time elapsed: 0.002 2025-03-14 08:18:23.159 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 8dbdfa8e-53d3-48ba-b0d4-a62eee529f3c. Wait time elapsed: 0.002 2025-03-14 08:18:23.159 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d52aeaf3-38f6-436e-ae4e-6ea09abb2ce8. Wait time elapsed: 0.002 2025-03-14 08:18:23.159 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d52aeaf3-38f6-436e-ae4e-6ea09abb2ce8. Wait time elapsed: 0.002 2025-03-14 08:18:23.160 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 7cbab367-a849-4694-9b27-fef5f1a9cbaf. Wait time elapsed: 0.003 2025-03-14 08:18:23.160 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 7cbab367-a849-4694-9b27-fef5f1a9cbaf. Wait time elapsed: 0.003 2025-03-14 08:18:24.100 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d52aeaf3-38f6-436e-ae4e-6ea09abb2ce8. Time elapsed: 0.941 2025-03-14 08:18:24.100 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id f261fda9-4b09-4f36-8611-bdf283c8663f. Wait time elapsed: 0.816 2025-03-14 08:18:24.101 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id f261fda9-4b09-4f36-8611-bdf283c8663f. Wait time elapsed: 0.816 2025-03-14 08:18:24.536 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 8dbdfa8e-53d3-48ba-b0d4-a62eee529f3c. Time elapsed: 1.377 2025-03-14 08:18:24.536 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 936c5c32-fa73-427b-aca2-7f4985a6cec8. Wait time elapsed: 1.252 2025-03-14 08:18:24.536 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 936c5c32-fa73-427b-aca2-7f4985a6cec8. Wait time elapsed: 1.252 2025-03-14 08:18:24.588 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 7cbab367-a849-4694-9b27-fef5f1a9cbaf. Time elapsed: 1.428 2025-03-14 08:18:24.588 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 174c359b-bc85-42e2-9ff0-50be7bc5116b. Wait time elapsed: 0.592 2025-03-14 08:18:24.588 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 174c359b-bc85-42e2-9ff0-50be7bc5116b. Wait time elapsed: 0.592 2025-03-14 08:18:24.632 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id dfccaa46-5671-4a30-acfb-b64b87580eff. Time elapsed: 1.475 2025-03-14 08:18:24.632 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 1db8b183-984d-4719-8e5b-75d6e0cec604. Wait time elapsed: 0.817 2025-03-14 08:18:24.632 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 1db8b183-984d-4719-8e5b-75d6e0cec604. Wait time elapsed: 0.817 2025-03-14 08:18:24.982 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id f261fda9-4b09-4f36-8611-bdf283c8663f. Time elapsed: 0.882 2025-03-14 08:18:24.982 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 5c2e7ab6-58e0-4c31-aa5f-60968c052135. Wait time elapsed: 0.351 2025-03-14 08:18:24.983 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 5c2e7ab6-58e0-4c31-aa5f-60968c052135. Wait time elapsed: 0.351 2025-03-14 08:18:26.036 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 174c359b-bc85-42e2-9ff0-50be7bc5116b. Time elapsed: 1.448 2025-03-14 08:18:26.036 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 5389d23d-4774-48da-88b1-7d14201e9957. Wait time elapsed: 0.515 2025-03-14 08:18:26.036 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 5389d23d-4774-48da-88b1-7d14201e9957. Wait time elapsed: 0.515 2025-03-14 08:18:26.095 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 936c5c32-fa73-427b-aca2-7f4985a6cec8. Time elapsed: 1.559 2025-03-14 08:18:26.095 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id fb85b66c-0643-4416-915d-df87c43978b9. Wait time elapsed: 1.508 2025-03-14 08:18:26.096 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id fb85b66c-0643-4416-915d-df87c43978b9. Wait time elapsed: 1.508 2025-03-14 08:18:26.128 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 5c2e7ab6-58e0-4c31-aa5f-60968c052135. Time elapsed: 1.145 2025-03-14 08:18:26.128 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 3c2c7b1d-2563-4f88-9be9-699ce87da5fd. Wait time elapsed: 0.888 2025-03-14 08:18:26.128 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 3c2c7b1d-2563-4f88-9be9-699ce87da5fd. Wait time elapsed: 0.888 2025-03-14 08:18:26.182 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 1db8b183-984d-4719-8e5b-75d6e0cec604. Time elapsed: 1.550 2025-03-14 08:18:26.183 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 442bfcb7-6d6f-4da9-9173-5f974b229e69. Wait time elapsed: 1.440 2025-03-14 08:18:26.183 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 442bfcb7-6d6f-4da9-9173-5f974b229e69. Wait time elapsed: 1.440 2025-03-14 08:18:27.225 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 3c2c7b1d-2563-4f88-9be9-699ce87da5fd. Time elapsed: 1.097 2025-03-14 08:18:27.270 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 5389d23d-4774-48da-88b1-7d14201e9957. Time elapsed: 1.234 2025-03-14 08:18:27.393 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id fb85b66c-0643-4416-915d-df87c43978b9. Time elapsed: 1.298 2025-03-14 08:18:27.494 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 442bfcb7-6d6f-4da9-9173-5f974b229e69. Time elapsed: 1.311 2025-03-14 08:18:27.494 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 5f4b6687-a78f-4bd0-94c7-726c56de6ca9. Wait time elapsed: 0.717 2025-03-14 08:18:27.494 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 5f4b6687-a78f-4bd0-94c7-726c56de6ca9. Wait time elapsed: 0.717 2025-03-14 08:18:28.768 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 5f4b6687-a78f-4bd0-94c7-726c56de6ca9. Time elapsed: 1.274 2025-03-14 08:19:19.337 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id ee7583f4-20a1-460a-aab8-f2caace54aa2. Wait time elapsed: 0.001 2025-03-14 08:19:19.337 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id ee7583f4-20a1-460a-aab8-f2caace54aa2. Wait time elapsed: 0.001 2025-03-14 08:19:19.340 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id fdfdc032-4eb8-4507-b12e-9ea62100893c. Wait time elapsed: 0.001 2025-03-14 08:19:19.340 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id fdfdc032-4eb8-4507-b12e-9ea62100893c. Wait time elapsed: 0.001 2025-03-14 08:19:19.341 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 21ee7991-e406-46f3-abe3-2614f837e41b. Wait time elapsed: 0.000 2025-03-14 08:19:19.341 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 21ee7991-e406-46f3-abe3-2614f837e41b. Wait time elapsed: 0.001 2025-03-14 08:19:19.341 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 68f23740-b6c2-44e9-acec-8c0b585ad308. Wait time elapsed: 0.001 2025-03-14 08:19:19.342 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 68f23740-b6c2-44e9-acec-8c0b585ad308. Wait time elapsed: 0.001 2025-03-14 08:19:20.132 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id fdfdc032-4eb8-4507-b12e-9ea62100893c. Time elapsed: 0.792 2025-03-14 08:19:20.132 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 360b8b36-337c-42a8-83f4-af223a17c589. Wait time elapsed: 0.788 2025-03-14 08:19:20.132 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 360b8b36-337c-42a8-83f4-af223a17c589. Wait time elapsed: 0.788 2025-03-14 08:19:20.132 2013 WARNING neutron.agent.l3.agent [-] Info for router ebd75b85-e4e9-4c99-9966-17a1ea1a3b14 was not found. Performing router cleanup 2025-03-14 08:19:20.133 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 360b8b36-337c-42a8-83f4-af223a17c589. Time elapsed: 0.001 2025-03-14 08:19:20.314 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id ee7583f4-20a1-460a-aab8-f2caace54aa2. Time elapsed: 0.977 2025-03-14 08:19:20.314 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id cf7796fb-c00e-4b81-90a1-c9673ad8840f. Wait time elapsed: 0.969 2025-03-14 08:19:20.314 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id cf7796fb-c00e-4b81-90a1-c9673ad8840f. Wait time elapsed: 0.969 2025-03-14 08:19:20.314 2013 WARNING neutron.agent.l3.agent [-] Info for router 668e4a57-9041-407f-b723-a64936d4be42 was not found. Performing router cleanup 2025-03-14 08:19:20.315 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id cf7796fb-c00e-4b81-90a1-c9673ad8840f. Time elapsed: 0.001 2025-03-14 08:19:20.340 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 21ee7991-e406-46f3-abe3-2614f837e41b. Time elapsed: 0.999 2025-03-14 08:19:20.340 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id a6599765-e38d-4c24-98be-4c538dc74de1. Wait time elapsed: 0.995 2025-03-14 08:19:20.341 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id a6599765-e38d-4c24-98be-4c538dc74de1. Wait time elapsed: 0.995 2025-03-14 08:19:20.341 2013 WARNING neutron.agent.l3.agent [-] Info for router e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3 was not found. Performing router cleanup 2025-03-14 08:19:20.341 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id a6599765-e38d-4c24-98be-4c538dc74de1. Time elapsed: 0.001 2025-03-14 08:19:20.386 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 68f23740-b6c2-44e9-acec-8c0b585ad308. Time elapsed: 1.045 2025-03-14 08:19:20.386 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 3931fe79-c1d0-451e-b361-25ea0943fc00. Wait time elapsed: 1.040 2025-03-14 08:19:20.387 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 3931fe79-c1d0-451e-b361-25ea0943fc00. Wait time elapsed: 1.040 2025-03-14 08:19:20.387 2013 WARNING neutron.agent.l3.agent [-] Info for router 4b89508c-beaf-43cf-92a6-9e080c03554d was not found. Performing router cleanup 2025-03-14 08:19:20.387 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 3931fe79-c1d0-451e-b361-25ea0943fc00. Time elapsed: 0.001 2025-03-14 08:22:34.904 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 5f2a7310-1397-4204-bc34-9dbf4b6a414b. Wait time elapsed: 0.000 2025-03-14 08:22:34.904 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 5f2a7310-1397-4204-bc34-9dbf4b6a414b. Wait time elapsed: 0.001 2025-03-14 08:22:34.905 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 09619b3a-5f30-407e-bd15-37cb69fc744d. Wait time elapsed: 0.001 2025-03-14 08:22:34.905 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 09619b3a-5f30-407e-bd15-37cb69fc744d. Wait time elapsed: 0.001 2025-03-14 08:22:34.906 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 228417a8-bf70-40d1-839a-8920f1e7ef3e. Wait time elapsed: 0.002 2025-03-14 08:22:34.906 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 228417a8-bf70-40d1-839a-8920f1e7ef3e. Wait time elapsed: 0.002 2025-03-14 08:22:34.907 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 14770839-ba6d-4031-a5da-c4f9ecf5b917. Wait time elapsed: 0.003 2025-03-14 08:22:34.907 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 14770839-ba6d-4031-a5da-c4f9ecf5b917. Wait time elapsed: 0.003 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.432 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:22:35.433 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 3381e31f-1796-48da-ac1a-fdde94d72897. Wait time elapsed: 0.398 2025-03-14 08:22:35.433 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 3381e31f-1796-48da-ac1a-fdde94d72897. Wait time elapsed: 0.398 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.440 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:22:35.441 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 54461540-a5ee-4b0e-986a-d27fd8dfd58d. Wait time elapsed: 0.406 2025-03-14 08:22:35.441 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 54461540-a5ee-4b0e-986a-d27fd8dfd58d. Wait time elapsed: 0.406 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'f269cf3d-8a00-4b5b-99c2-63cae0968f42\', \'host\': \'hv-017\', \'router_id\': \'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'f269cf3d-8a00-4b5b-99c2-63cae0968f42\', \'host\': \'hv-017\', \'router_id\': \'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017' for key 'PRIMARY'") 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': 'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'hv-017', 'router_id': 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'f269cf3d-8a00-4b5b-99c2-63cae0968f42\', \'host\': \'hv-017\', \'router_id\': \'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'f269cf3d-8a00-4b5b-99c2-63cae0968f42-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'f269cf3d-8a00-4b5b-99c2-63cae0968f42\', \'host\': \'hv-017\', \'router_id\': \'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:22:35.897 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:22:35.897 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 73c82544-4c30-4a2c-a399-31ca9dd3d9a5. Wait time elapsed: 0.050 2025-03-14 08:22:35.898 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 73c82544-4c30-4a2c-a399-31ca9dd3d9a5. Wait time elapsed: 0.051 2025-03-14 08:22:39.163 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 5f2a7310-1397-4204-bc34-9dbf4b6a414b. Time elapsed: 4.259 2025-03-14 08:22:39.163 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d627a179-861a-47eb-8255-047b2d6f5c10. Wait time elapsed: 4.129 2025-03-14 08:22:39.164 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d627a179-861a-47eb-8255-047b2d6f5c10. Wait time elapsed: 4.129 2025-03-14 08:22:40.467 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 09619b3a-5f30-407e-bd15-37cb69fc744d. Time elapsed: 5.562 2025-03-14 08:22:40.468 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id a148f6a8-4db0-47f5-81b3-990f8e4c958b. Wait time elapsed: 5.433 2025-03-14 08:22:40.468 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id a148f6a8-4db0-47f5-81b3-990f8e4c958b. Wait time elapsed: 5.433 2025-03-14 08:22:40.486 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d627a179-861a-47eb-8255-047b2d6f5c10. Time elapsed: 1.323 2025-03-14 08:22:41.747 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id a148f6a8-4db0-47f5-81b3-990f8e4c958b. Time elapsed: 1.279 2025-03-14 08:22:41.968 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 73c82544-4c30-4a2c-a399-31ca9dd3d9a5. Time elapsed: 6.070 2025-03-14 08:22:41.968 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 8cc5c7fc-8a39-43ed-8ad4-be9b02fdc1ea. Wait time elapsed: 5.554 2025-03-14 08:22:41.968 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 8cc5c7fc-8a39-43ed-8ad4-be9b02fdc1ea. Wait time elapsed: 5.554 2025-03-14 08:22:42.192 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 54461540-a5ee-4b0e-986a-d27fd8dfd58d. Time elapsed: 6.752 2025-03-14 08:22:42.193 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 55ac37f5-5c79-4fc4-94ec-1d0e28f437bf. Wait time elapsed: 5.779 2025-03-14 08:22:42.193 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 55ac37f5-5c79-4fc4-94ec-1d0e28f437bf. Wait time elapsed: 5.779 2025-03-14 08:22:42.194 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 758719bc-da4e-469f-82de-958bfb03c04a. Wait time elapsed: 0.002 2025-03-14 08:22:42.194 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 758719bc-da4e-469f-82de-958bfb03c04a. Wait time elapsed: 0.002 2025-03-14 08:22:43.178 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 8cc5c7fc-8a39-43ed-8ad4-be9b02fdc1ea. Time elapsed: 1.210 2025-03-14 08:22:43.178 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 2c9f34f8-a6d0-48be-9c45-c2b694e8f833. Wait time elapsed: 0.986 2025-03-14 08:22:43.178 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 2c9f34f8-a6d0-48be-9c45-c2b694e8f833. Wait time elapsed: 0.986 2025-03-14 08:22:43.399 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 758719bc-da4e-469f-82de-958bfb03c04a. Time elapsed: 1.205 2025-03-14 08:22:43.509 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 55ac37f5-5c79-4fc4-94ec-1d0e28f437bf. Time elapsed: 1.316 2025-03-14 08:22:44.491 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 2c9f34f8-a6d0-48be-9c45-c2b694e8f833. Time elapsed: 1.313 2025-03-14 08:23:43.953 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id dfb29f43-e328-41e7-afe1-9b36049260d7. Wait time elapsed: 0.000 2025-03-14 08:23:43.953 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id dfb29f43-e328-41e7-afe1-9b36049260d7. Wait time elapsed: 0.000 2025-03-14 08:23:43.957 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id d6975249-7603-4bd2-b4dd-20f0299c77d4. Wait time elapsed: 0.001 2025-03-14 08:23:43.957 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id d6975249-7603-4bd2-b4dd-20f0299c77d4. Wait time elapsed: 0.001 2025-03-14 08:23:43.959 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 3bfac320-d8cd-4ac4-a5a6-ac53bf24ace2. Wait time elapsed: 0.001 2025-03-14 08:23:43.959 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 3bfac320-d8cd-4ac4-a5a6-ac53bf24ace2. Wait time elapsed: 0.001 2025-03-14 08:23:43.960 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id daa516d3-ec50-494f-81ee-be2b983623df. Wait time elapsed: 0.000 2025-03-14 08:23:43.960 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id daa516d3-ec50-494f-81ee-be2b983623df. Wait time elapsed: 0.000 2025-03-14 08:23:44.618 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id dfb29f43-e328-41e7-afe1-9b36049260d7. Time elapsed: 0.665 2025-03-14 08:23:44.846 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id d6975249-7603-4bd2-b4dd-20f0299c77d4. Time elapsed: 0.889 2025-03-14 08:23:44.878 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 3bfac320-d8cd-4ac4-a5a6-ac53bf24ace2. Time elapsed: 0.920 2025-03-14 08:23:44.939 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id daa516d3-ec50-494f-81ee-be2b983623df. Time elapsed: 0.979 2025-03-14 08:25:03.170 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 4d732734-5128-4f55-bf84-d33b12debee6. Wait time elapsed: 0.000 2025-03-14 08:25:03.171 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 4d732734-5128-4f55-bf84-d33b12debee6. Wait time elapsed: 0.001 2025-03-14 08:25:03.172 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id c7422dec-9b8e-4e51-9fd3-f2c0fc19132f. Wait time elapsed: 0.002 2025-03-14 08:25:03.172 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id c7422dec-9b8e-4e51-9fd3-f2c0fc19132f. Wait time elapsed: 0.002 2025-03-14 08:25:03.173 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id f552c9e0-47ab-4f62-9cd1-5f9c685790a8. Wait time elapsed: 0.003 2025-03-14 08:25:03.173 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id f552c9e0-47ab-4f62-9cd1-5f9c685790a8. Wait time elapsed: 0.003 2025-03-14 08:25:03.174 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 0aee79ab-c8d0-449b-953d-13db53d232fb. Wait time elapsed: 0.004 2025-03-14 08:25:03.174 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 0aee79ab-c8d0-449b-953d-13db53d232fb. Wait time elapsed: 0.004 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:25:03.622 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:25:03.622 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 687451a4-074d-4603-820a-69a04f77c75d. Wait time elapsed: 0.331 2025-03-14 08:25:03.623 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 687451a4-074d-4603-820a-69a04f77c75d. Wait time elapsed: 0.331 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:25:03.752 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:25:03.752 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 71ba1fc0-b939-4fa6-8b34-08ade09ba2e2. Wait time elapsed: 0.460 2025-03-14 08:25:03.752 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 71ba1fc0-b939-4fa6-8b34-08ade09ba2e2. Wait time elapsed: 0.460 2025-03-14 08:25:04.367 2013 WARNING neutron.agent.l3.dvr_local_router [req-8bcb1869-7d39-4061-b5f3-a03ada103c85 - - - - -] Device qr-5be154bc-e7 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-14 08:25:08.460 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 4d732734-5128-4f55-bf84-d33b12debee6. Time elapsed: 5.290 2025-03-14 08:25:08.461 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d3d84dbe-9336-4892-8608-583428a3d509. Wait time elapsed: 5.169 2025-03-14 08:25:08.461 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d3d84dbe-9336-4892-8608-583428a3d509. Wait time elapsed: 5.169 2025-03-14 08:25:08.884 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id c7422dec-9b8e-4e51-9fd3-f2c0fc19132f. Time elapsed: 5.712 2025-03-14 08:25:08.884 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 6ebfe53a-7aa2-4ccd-a3ad-ff3e312c8d19. Wait time elapsed: 5.592 2025-03-14 08:25:08.884 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 6ebfe53a-7aa2-4ccd-a3ad-ff3e312c8d19. Wait time elapsed: 5.593 2025-03-14 08:25:09.474 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d3d84dbe-9336-4892-8608-583428a3d509. Time elapsed: 1.014 2025-03-14 08:25:10.359 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 71ba1fc0-b939-4fa6-8b34-08ade09ba2e2. Time elapsed: 6.607 2025-03-14 08:25:10.360 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e73dc392-93d3-4fd7-a90c-8ed4da0fb2f1. Wait time elapsed: 6.044 2025-03-14 08:25:10.360 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e73dc392-93d3-4fd7-a90c-8ed4da0fb2f1. Wait time elapsed: 6.044 2025-03-14 08:25:10.361 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id dabf8075-2ef9-40a3-bd14-d9077429baab. Wait time elapsed: 0.002 2025-03-14 08:25:10.361 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id dabf8075-2ef9-40a3-bd14-d9077429baab. Wait time elapsed: 0.002 2025-03-14 08:25:10.396 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 6ebfe53a-7aa2-4ccd-a3ad-ff3e312c8d19. Time elapsed: 1.512 2025-03-14 08:25:10.397 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id c1ea0b7c-b18e-4dc5-816c-5e9674c0e9bb. Wait time elapsed: 0.037 2025-03-14 08:25:10.397 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id c1ea0b7c-b18e-4dc5-816c-5e9674c0e9bb. Wait time elapsed: 0.038 2025-03-14 08:25:10.659 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 687451a4-074d-4603-820a-69a04f77c75d. Time elapsed: 7.037 2025-03-14 08:25:10.660 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id fa0ee9d6-8de6-47de-b17d-e443050251a9. Wait time elapsed: 6.147 2025-03-14 08:25:10.660 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id fa0ee9d6-8de6-47de-b17d-e443050251a9. Wait time elapsed: 6.148 2025-03-14 08:25:11.166 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id dabf8075-2ef9-40a3-bd14-d9077429baab. Time elapsed: 0.805 2025-03-14 08:25:11.560 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id e73dc392-93d3-4fd7-a90c-8ed4da0fb2f1. Time elapsed: 1.200 2025-03-14 08:25:11.560 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 6e195e2f-40f0-4233-aab5-040263077c30. Wait time elapsed: 0.901 2025-03-14 08:25:11.561 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 6e195e2f-40f0-4233-aab5-040263077c30. Wait time elapsed: 0.901 2025-03-14 08:25:11.924 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id c1ea0b7c-b18e-4dc5-816c-5e9674c0e9bb. Time elapsed: 1.528 2025-03-14 08:25:11.925 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 0b3cf819-7c8c-4fb3-9a68-7de14d2cdafe. Wait time elapsed: 1.265 2025-03-14 08:25:11.925 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 0b3cf819-7c8c-4fb3-9a68-7de14d2cdafe. Wait time elapsed: 1.265 2025-03-14 08:25:11.987 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id fa0ee9d6-8de6-47de-b17d-e443050251a9. Time elapsed: 1.327 2025-03-14 08:25:12.847 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 6e195e2f-40f0-4233-aab5-040263077c30. Time elapsed: 1.286 2025-03-14 08:25:12.969 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 0b3cf819-7c8c-4fb3-9a68-7de14d2cdafe. Time elapsed: 1.044 2025-03-14 08:25:49.736 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id f62d846c-0a0d-4be1-acfa-ababeab9188d. Wait time elapsed: 0.001 2025-03-14 08:25:49.737 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id f62d846c-0a0d-4be1-acfa-ababeab9188d. Wait time elapsed: 0.001 2025-03-14 08:25:49.738 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 767e62f5-0f4d-467f-9ee9-b0836fb8cf1e. Wait time elapsed: 0.001 2025-03-14 08:25:49.739 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 767e62f5-0f4d-467f-9ee9-b0836fb8cf1e. Wait time elapsed: 0.001 2025-03-14 08:25:49.739 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 6665992b-f1e9-466b-bdeb-881bc20709dc. Wait time elapsed: 0.000 2025-03-14 08:25:49.740 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 6665992b-f1e9-466b-bdeb-881bc20709dc. Wait time elapsed: 0.000 2025-03-14 08:25:49.740 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 9773c0da-4a28-4c42-adb5-5b9617ff4b88. Wait time elapsed: 0.001 2025-03-14 08:25:49.740 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 9773c0da-4a28-4c42-adb5-5b9617ff4b88. Wait time elapsed: 0.001 2025-03-14 08:25:50.451 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id f62d846c-0a0d-4be1-acfa-ababeab9188d. Time elapsed: 0.715 2025-03-14 08:25:50.712 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 6665992b-f1e9-466b-bdeb-881bc20709dc. Time elapsed: 0.972 2025-03-14 08:25:50.760 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 767e62f5-0f4d-467f-9ee9-b0836fb8cf1e. Time elapsed: 1.022 2025-03-14 08:25:50.816 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 9773c0da-4a28-4c42-adb5-5b9617ff4b88. Time elapsed: 1.076 2025-03-14 08:27:46.356 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ce33ef75-f196-41b9-a82c-9b850a5c3755. Wait time elapsed: 0.000 2025-03-14 08:27:46.356 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ce33ef75-f196-41b9-a82c-9b850a5c3755. Wait time elapsed: 0.001 2025-03-14 08:27:46.357 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id d135404e-9bf7-42df-9aca-dca8890b7ba7. Wait time elapsed: 0.001 2025-03-14 08:27:46.357 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id d135404e-9bf7-42df-9aca-dca8890b7ba7. Wait time elapsed: 0.001 2025-03-14 08:27:46.358 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 2ecd4e5a-8ede-471a-83d5-595d3eda716e. Wait time elapsed: 0.002 2025-03-14 08:27:46.358 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 2ecd4e5a-8ede-471a-83d5-595d3eda716e. Wait time elapsed: 0.002 2025-03-14 08:27:46.359 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id ae9e81dd-832e-4c3f-8acf-b862e3d2c275. Wait time elapsed: 0.003 2025-03-14 08:27:46.359 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id ae9e81dd-832e-4c3f-8acf-b862e3d2c275. Wait time elapsed: 0.003 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:27:46.825 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:27:46.826 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 2b6635ba-dfbe-457f-a540-700d2c08cb37. Wait time elapsed: 0.347 2025-03-14 08:27:46.826 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 2b6635ba-dfbe-457f-a540-700d2c08cb37. Wait time elapsed: 0.347 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:27:46.913 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:27:46.913 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 0e86cea6-5f6c-41d0-9ceb-3eeb7e9f7f82. Wait time elapsed: 0.435 2025-03-14 08:27:46.913 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 0e86cea6-5f6c-41d0-9ceb-3eeb7e9f7f82. Wait time elapsed: 0.435 2025-03-14 08:27:51.277 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id ce33ef75-f196-41b9-a82c-9b850a5c3755. Time elapsed: 4.921 2025-03-14 08:27:51.277 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 836557e0-45be-4b41-ac12-e1dae9f9b5e1. Wait time elapsed: 4.799 2025-03-14 08:27:51.277 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 836557e0-45be-4b41-ac12-e1dae9f9b5e1. Wait time elapsed: 4.799 2025-03-14 08:27:51.918 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id d135404e-9bf7-42df-9aca-dca8890b7ba7. Time elapsed: 5.561 2025-03-14 08:27:51.918 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 2506cb21-1253-4a7f-906f-51243736f2bd. Wait time elapsed: 5.440 2025-03-14 08:27:51.918 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 2506cb21-1253-4a7f-906f-51243736f2bd. Wait time elapsed: 5.440 2025-03-14 08:27:52.422 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 836557e0-45be-4b41-ac12-e1dae9f9b5e1. Time elapsed: 1.145 2025-03-14 08:27:53.472 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 2b6635ba-dfbe-457f-a540-700d2c08cb37. Time elapsed: 6.646 2025-03-14 08:27:53.472 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id af958dea-0b93-42e3-b80b-ac3a62859491. Wait time elapsed: 6.008 2025-03-14 08:27:53.472 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id af958dea-0b93-42e3-b80b-ac3a62859491. Wait time elapsed: 6.008 2025-03-14 08:27:53.473 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 43bb6161-c07d-4edd-bef6-b78e476c1559. Wait time elapsed: 0.002 2025-03-14 08:27:53.474 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 43bb6161-c07d-4edd-bef6-b78e476c1559. Wait time elapsed: 0.002 2025-03-14 08:27:53.552 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 2506cb21-1253-4a7f-906f-51243736f2bd. Time elapsed: 1.634 2025-03-14 08:27:53.553 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 6ba18363-75e5-476d-a781-08ab42155d32. Wait time elapsed: 0.081 2025-03-14 08:27:53.553 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 6ba18363-75e5-476d-a781-08ab42155d32. Wait time elapsed: 0.081 2025-03-14 08:27:53.693 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 0e86cea6-5f6c-41d0-9ceb-3eeb7e9f7f82. Time elapsed: 6.780 2025-03-14 08:27:53.694 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id b20d3640-e0e6-4aa3-a493-b0c6f276f62f. Wait time elapsed: 5.865 2025-03-14 08:27:53.694 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id b20d3640-e0e6-4aa3-a493-b0c6f276f62f. Wait time elapsed: 5.866 2025-03-14 08:27:54.423 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 43bb6161-c07d-4edd-bef6-b78e476c1559. Time elapsed: 0.950 2025-03-14 08:27:54.739 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 6ba18363-75e5-476d-a781-08ab42155d32. Time elapsed: 1.186 2025-03-14 08:27:54.739 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 78256522-4933-4c76-bd7b-2e5c28c6f14a. Wait time elapsed: 1.046 2025-03-14 08:27:54.739 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 78256522-4933-4c76-bd7b-2e5c28c6f14a. Wait time elapsed: 1.046 2025-03-14 08:27:54.892 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id b20d3640-e0e6-4aa3-a493-b0c6f276f62f. Time elapsed: 1.198 2025-03-14 08:27:55.044 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id af958dea-0b93-42e3-b80b-ac3a62859491. Time elapsed: 1.572 2025-03-14 08:27:55.045 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 0c0ff7af-b916-45f8-af84-f888b74b7497. Wait time elapsed: 1.352 2025-03-14 08:27:55.045 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 0c0ff7af-b916-45f8-af84-f888b74b7497. Wait time elapsed: 1.352 2025-03-14 08:27:55.894 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 78256522-4933-4c76-bd7b-2e5c28c6f14a. Time elapsed: 1.155 2025-03-14 08:27:56.451 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 0c0ff7af-b916-45f8-af84-f888b74b7497. Time elapsed: 1.406 2025-03-14 08:28:27.116 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 0e7effd0-302f-4cf4-b0b6-3283f89059e2. Wait time elapsed: 0.001 2025-03-14 08:28:27.117 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 0e7effd0-302f-4cf4-b0b6-3283f89059e2. Wait time elapsed: 0.001 2025-03-14 08:28:27.117 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 871562aa-96de-4313-b944-abbd8eb9a3fb. Wait time elapsed: 0.001 2025-03-14 08:28:27.118 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 871562aa-96de-4313-b944-abbd8eb9a3fb. Wait time elapsed: 0.002 2025-03-14 08:28:27.119 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 0c324ba7-5988-4851-8a90-a4335eea7583. Wait time elapsed: 0.001 2025-03-14 08:28:27.119 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 0c324ba7-5988-4851-8a90-a4335eea7583. Wait time elapsed: 0.001 2025-03-14 08:28:27.120 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 5454cbb2-047a-4ea9-b739-283c859fdb99. Wait time elapsed: 0.000 2025-03-14 08:28:27.120 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id 5454cbb2-047a-4ea9-b739-283c859fdb99. Wait time elapsed: 0.000 2025-03-14 08:28:28.034 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 871562aa-96de-4313-b944-abbd8eb9a3fb. Time elapsed: 0.916 2025-03-14 08:28:28.201 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 0c324ba7-5988-4851-8a90-a4335eea7583. Time elapsed: 1.082 2025-03-14 08:28:28.206 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 0e7effd0-302f-4cf4-b0b6-3283f89059e2. Time elapsed: 1.090 2025-03-14 08:28:28.223 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 5454cbb2-047a-4ea9-b739-283c859fdb99. Time elapsed: 1.103 2025-03-14 08:36:42.139 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 8e95f348-6278-4756-bc45-b650f062b63a. Wait time elapsed: 0.000 2025-03-14 08:36:42.140 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 8e95f348-6278-4756-bc45-b650f062b63a. Wait time elapsed: 0.001 2025-03-14 08:36:42.140 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 63e130e9-54fc-4775-812a-2c01cfc80e67. Wait time elapsed: 0.001 2025-03-14 08:36:42.140 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 63e130e9-54fc-4775-812a-2c01cfc80e67. Wait time elapsed: 0.001 2025-03-14 08:36:42.141 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 5109ec4e-f3bf-496d-a80a-647878cc9f1d. Wait time elapsed: 0.002 2025-03-14 08:36:42.141 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 5109ec4e-f3bf-496d-a80a-647878cc9f1d. Wait time elapsed: 0.002 2025-03-14 08:36:42.142 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 775b1cc6-676b-45ec-9c47-11464c13f636. Wait time elapsed: 0.003 2025-03-14 08:36:42.142 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 775b1cc6-676b-45ec-9c47-11464c13f636. Wait time elapsed: 0.003 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:36:42.657 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:36:42.658 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 242f70af-722d-42b9-b629-b2ca6cab20ea. Wait time elapsed: 0.391 2025-03-14 08:36:42.658 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 242f70af-722d-42b9-b629-b2ca6cab20ea. Wait time elapsed: 0.391 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:36:42.665 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:36:42.665 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 95ad0084-294e-4c7b-8d59-39bd027e0bc7. Wait time elapsed: 0.399 2025-03-14 08:36:42.666 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 95ad0084-294e-4c7b-8d59-39bd027e0bc7. Wait time elapsed: 0.399 2025-03-14 08:36:43.717 2013 WARNING neutron.agent.l3.dvr_local_router [req-eb716f80-7db4-4ab8-8597-989bd0103201 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-8e2c27c0-75 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-14 08:36:47.245 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 8e95f348-6278-4756-bc45-b650f062b63a. Time elapsed: 5.106 2025-03-14 08:36:47.246 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 2c54de21-a948-422c-bf08-5a983d6eaaad. Wait time elapsed: 4.979 2025-03-14 08:36:47.246 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 2c54de21-a948-422c-bf08-5a983d6eaaad. Wait time elapsed: 4.979 2025-03-14 08:36:48.089 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 95ad0084-294e-4c7b-8d59-39bd027e0bc7. Time elapsed: 5.424 2025-03-14 08:36:48.090 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 7b9d127a-eef9-4212-b5fb-231fb3befdd3. Wait time elapsed: 4.406 2025-03-14 08:36:48.090 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 7b9d127a-eef9-4212-b5fb-231fb3befdd3. Wait time elapsed: 4.406 2025-03-14 08:36:48.367 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 2c54de21-a948-422c-bf08-5a983d6eaaad. Time elapsed: 1.121 2025-03-14 08:36:49.023 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 242f70af-722d-42b9-b629-b2ca6cab20ea. Time elapsed: 6.365 2025-03-14 08:36:49.023 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id c2794cb6-738d-4267-8276-3aead99fc3c2. Wait time elapsed: 5.855 2025-03-14 08:36:49.024 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id c2794cb6-738d-4267-8276-3aead99fc3c2. Wait time elapsed: 5.855 2025-03-14 08:36:49.025 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 3a38af36-aa5f-4ca5-bb23-d26e95bbabc2. Wait time elapsed: 0.002 2025-03-14 08:36:49.025 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 3a38af36-aa5f-4ca5-bb23-d26e95bbabc2. Wait time elapsed: 0.002 2025-03-14 08:36:49.331 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 775b1cc6-676b-45ec-9c47-11464c13f636. Time elapsed: 7.189 2025-03-14 08:36:49.332 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 16008099-b2b2-4c23-8d35-bd984930c808. Wait time elapsed: 5.681 2025-03-14 08:36:49.332 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 16008099-b2b2-4c23-8d35-bd984930c808. Wait time elapsed: 5.682 2025-03-14 08:36:49.476 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 7b9d127a-eef9-4212-b5fb-231fb3befdd3. Time elapsed: 1.386 2025-03-14 08:36:49.477 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id cb68d643-0e77-4ddf-9f2c-cbc25143ddff. Wait time elapsed: 0.454 2025-03-14 08:36:49.477 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id cb68d643-0e77-4ddf-9f2c-cbc25143ddff. Wait time elapsed: 0.454 2025-03-14 08:36:49.871 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 3a38af36-aa5f-4ca5-bb23-d26e95bbabc2. Time elapsed: 0.846 2025-03-14 08:36:50.598 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 16008099-b2b2-4c23-8d35-bd984930c808. Time elapsed: 1.267 2025-03-14 08:36:50.685 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id cb68d643-0e77-4ddf-9f2c-cbc25143ddff. Time elapsed: 1.208 2025-03-14 08:36:50.747 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id c2794cb6-738d-4267-8276-3aead99fc3c2. Time elapsed: 1.724 2025-03-14 08:36:50.748 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e368b3e4-cb84-46e2-85ae-65026d00cb88. Wait time elapsed: 1.553 2025-03-14 08:36:50.748 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id e368b3e4-cb84-46e2-85ae-65026d00cb88. Wait time elapsed: 1.554 2025-03-14 08:36:51.942 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id e368b3e4-cb84-46e2-85ae-65026d00cb88. Time elapsed: 1.194 2025-03-14 08:45:10.066 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 70d74b02-e6f4-4e5e-8dd0-4754f49d0180. Wait time elapsed: 0.001 2025-03-14 08:45:10.066 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 70d74b02-e6f4-4e5e-8dd0-4754f49d0180. Wait time elapsed: 0.001 2025-03-14 08:45:10.067 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 093ab394-6bf0-42ef-bf7f-237af494aca7. Wait time elapsed: 0.002 2025-03-14 08:45:10.067 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 093ab394-6bf0-42ef-bf7f-237af494aca7. Wait time elapsed: 0.002 2025-03-14 08:45:10.068 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 1911a2f3-9ce1-4626-80a4-680d3186106a. Wait time elapsed: 0.001 2025-03-14 08:45:10.069 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 1911a2f3-9ce1-4626-80a4-680d3186106a. Wait time elapsed: 0.001 2025-03-14 08:45:10.070 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id e7a2c1af-d129-4888-818b-25e7aab92380. Wait time elapsed: 0.000 2025-03-14 08:45:10.070 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id e7a2c1af-d129-4888-818b-25e7aab92380. Wait time elapsed: 0.000 2025-03-14 08:45:10.949 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 093ab394-6bf0-42ef-bf7f-237af494aca7. Time elapsed: 0.882 2025-03-14 08:45:11.083 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 70d74b02-e6f4-4e5e-8dd0-4754f49d0180. Time elapsed: 1.017 2025-03-14 08:45:11.152 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 1911a2f3-9ce1-4626-80a4-680d3186106a. Time elapsed: 1.083 2025-03-14 08:45:11.166 2013 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id e7a2c1af-d129-4888-818b-25e7aab92380. Time elapsed: 1.096 2025-03-14 08:50:23.544 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 38c4400d-2900-42b4-9584-62c13f80ab4b. Wait time elapsed: 0.000 2025-03-14 08:50:23.544 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id 38c4400d-2900-42b4-9584-62c13f80ab4b. Wait time elapsed: 0.000 2025-03-14 08:50:23.545 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id e491b7fe-2710-4bba-a185-31d9b30d7a02. Wait time elapsed: 0.001 2025-03-14 08:50:23.545 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id e491b7fe-2710-4bba-a185-31d9b30d7a02. Wait time elapsed: 0.001 2025-03-14 08:50:23.546 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id a96c67c7-870c-4dbf-a85c-9288fdd54b2e. Wait time elapsed: 0.002 2025-03-14 08:50:23.546 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id a96c67c7-870c-4dbf-a85c-9288fdd54b2e. Wait time elapsed: 0.002 2025-03-14 08:50:23.547 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 4230cedc-dac7-4b4f-9a1b-b18e9eba9b01. Wait time elapsed: 0.003 2025-03-14 08:50:23.547 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 4230cedc-dac7-4b4f-9a1b-b18e9eba9b01. Wait time elapsed: 0.003 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:50:24.083 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:50:24.083 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 2b7e4020-0e9f-4183-9356-e1e3a20ec5ec. Wait time elapsed: 0.393 2025-03-14 08:50:24.083 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 2b7e4020-0e9f-4183-9356-e1e3a20ec5ec. Wait time elapsed: 0.393 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '4b89508c-beaf-43cf-92a6-9e080c03554d': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent raise result 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017' for key 'PRIMARY'") 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent [parameters: {'port_id': '84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'84c36ac1-9b0e-49e6-95b8-d1ee4bd73a90\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 08:50:24.146 2013 ERROR neutron.agent.l3.agent 2025-03-14 08:50:24.147 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 598b49da-962b-491a-bdc4-d3b9dca33131. Wait time elapsed: 0.456 2025-03-14 08:50:24.147 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 598b49da-962b-491a-bdc4-d3b9dca33131. Wait time elapsed: 0.456 2025-03-14 08:50:28.461 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 38c4400d-2900-42b4-9584-62c13f80ab4b. Time elapsed: 4.917 2025-03-14 08:50:28.462 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id e73ee0d4-6859-4f20-bbea-abcef3da33b8. Wait time elapsed: 4.771 2025-03-14 08:50:28.462 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id e73ee0d4-6859-4f20-bbea-abcef3da33b8. Wait time elapsed: 4.771 2025-03-14 08:50:29.018 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 2b7e4020-0e9f-4183-9356-e1e3a20ec5ec. Time elapsed: 4.935 2025-03-14 08:50:29.018 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 0aac4513-6757-490a-9550-903c4b1dba37. Wait time elapsed: 3.431 2025-03-14 08:50:29.018 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 0aac4513-6757-490a-9550-903c4b1dba37. Wait time elapsed: 3.431 2025-03-14 08:50:29.779 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id e73ee0d4-6859-4f20-bbea-abcef3da33b8. Time elapsed: 1.318 2025-03-14 08:50:30.165 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id a96c67c7-870c-4dbf-a85c-9288fdd54b2e. Time elapsed: 6.619 2025-03-14 08:50:30.166 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 01698fc3-cbed-47a4-b9ef-0503430508f5. Wait time elapsed: 5.462 2025-03-14 08:50:30.166 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 01698fc3-cbed-47a4-b9ef-0503430508f5. Wait time elapsed: 5.462 2025-03-14 08:50:30.167 2013 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id af44b09d-e1d4-446f-805b-d9b41604d7d2. Wait time elapsed: 0.001 2025-03-14 08:50:30.167 2013 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id af44b09d-e1d4-446f-805b-d9b41604d7d2. Wait time elapsed: 0.002 2025-03-14 08:50:30.442 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 0aac4513-6757-490a-9550-903c4b1dba37. Time elapsed: 1.423 2025-03-14 08:50:30.442 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 78fe5e37-2d0a-4ff7-82af-a925e1357ab4. Wait time elapsed: 0.277 2025-03-14 08:50:30.442 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 78fe5e37-2d0a-4ff7-82af-a925e1357ab4. Wait time elapsed: 0.277 2025-03-14 08:50:30.906 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 598b49da-962b-491a-bdc4-d3b9dca33131. Time elapsed: 6.759 2025-03-14 08:50:30.906 2013 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 803bc1f0-5d63-41db-bb17-659a5dac5201. Wait time elapsed: 6.000 2025-03-14 08:50:30.906 2013 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 803bc1f0-5d63-41db-bb17-659a5dac5201. Wait time elapsed: 6.000 2025-03-14 08:50:31.031 2013 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id af44b09d-e1d4-446f-805b-d9b41604d7d2. Time elapsed: 0.864 2025-03-14 08:50:31.416 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 01698fc3-cbed-47a4-b9ef-0503430508f5. Time elapsed: 1.250 2025-03-14 08:50:31.416 2013 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 25c7831b-a422-4206-969e-7ec76d190044. Wait time elapsed: 0.511 2025-03-14 08:50:31.416 2013 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 25c7831b-a422-4206-969e-7ec76d190044. Wait time elapsed: 0.511 2025-03-14 08:50:31.572 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 78fe5e37-2d0a-4ff7-82af-a925e1357ab4. Time elapsed: 1.130 2025-03-14 08:50:31.572 2013 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id fb160da6-9b60-43bf-b4a7-52aea2e4bfd1. Wait time elapsed: 0.666 2025-03-14 08:50:31.572 2013 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id fb160da6-9b60-43bf-b4a7-52aea2e4bfd1. Wait time elapsed: 0.667 2025-03-14 08:50:32.130 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 803bc1f0-5d63-41db-bb17-659a5dac5201. Time elapsed: 1.224 2025-03-14 08:50:32.918 2013 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 25c7831b-a422-4206-969e-7ec76d190044. Time elapsed: 1.502 2025-03-14 08:50:32.974 2013 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id fb160da6-9b60-43bf-b4a7-52aea2e4bfd1. Time elapsed: 1.402 2025-03-14 11:02:39.131 2013 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored 2025-03-14 11:02:39.132 2013 INFO neutron.agent.l3.agent [-] Stopping L3 agent 2025-03-14 11:02:39.133 2013 INFO oslo_rootwrap.client [req-93cf1bb5-b18e-4487-b084-131dd0a5d838 - - - - -] Stopping rootwrap daemon process with pid=314578 2025-03-14 11:02:41.344 385477 INFO neutron.common.config [-] Logging enabled! 2025-03-14 11:02:41.345 385477 INFO neutron.common.config [-] /usr/bin/neutron-l3-agent version 20.5.0 2025-03-14 11:02:41.702 385477 INFO neutron.agent.l3.agent [req-6607b78f-d15c-4028-a68f-98a5109da264 - - - - -] Agent HA routers count 0 2025-03-14 11:02:41.706 385477 INFO neutron.agent.agent_extensions_manager [req-6607b78f-d15c-4028-a68f-98a5109da264 - - - - -] Loaded agent extensions: ['fip_qos'] 2025-03-14 11:02:41.707 385477 INFO neutron.agent.agent_extensions_manager [req-6607b78f-d15c-4028-a68f-98a5109da264 - - - - -] Initializing agent extension 'fip_qos' 2025-03-14 11:02:41.738 385477 WARNING oslo_config.cfg [-] Deprecated: Option "allow_overlapping_ips" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. 2025-03-14 11:02:41.744 385477 INFO eventlet.wsgi.server [-] (385477) wsgi starting up on http:/var/lib/neutron/keepalived-state-change 2025-03-14 11:02:41.786 385477 INFO neutron.agent.l3.agent [-] L3 agent started 2025-03-14 11:02:43.789 385477 INFO oslo.privsep.daemon [req-f4375662-d044-4f84-8586-ff8e23f2947e - - - - -] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.namespace_cmd', '--privsep_sock_path', '/tmp/tmpgiw3eh5z/privsep.sock'] 2025-03-14 11:02:44.399 385477 INFO oslo.privsep.daemon [req-f4375662-d044-4f84-8586-ff8e23f2947e - - - - -] Spawned new privsep daemon via rootwrap 2025-03-14 11:02:44.298 385579 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-14 11:02:44.301 385579 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-14 11:02:44.302 385579 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none 2025-03-14 11:02:44.303 385579 INFO oslo.privsep.daemon [-] privsep daemon running as pid 385579 2025-03-14 11:02:46.376 385477 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 2, update_id 19e80549-a594-4f21-848b-1adb6636a2f3. Wait time elapsed: 0.000 2025-03-14 11:02:46.376 385477 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 2, update_id 19e80549-a594-4f21-848b-1adb6636a2f3. Wait time elapsed: 0.001 2025-03-14 11:02:46.377 385477 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp_o7yy09h/privsep.sock'] 2025-03-14 11:02:46.378 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 2, update_id 5f780d66-8862-4d47-bd71-770d0682b569. Wait time elapsed: 0.002 2025-03-14 11:02:46.378 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 2, update_id 5f780d66-8862-4d47-bd71-770d0682b569. Wait time elapsed: 0.002 2025-03-14 11:02:46.378 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 2, update_id 74004c09-a8ad-43f1-a30b-6a199148477c. Wait time elapsed: 0.003 2025-03-14 11:02:46.379 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 2, update_id 74004c09-a8ad-43f1-a30b-6a199148477c. Wait time elapsed: 0.003 2025-03-14 11:02:46.379 385477 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 2, update_id 15cf09e4-437c-4f1d-a624-fcd392d2e51d. Wait time elapsed: 0.003 2025-03-14 11:02:46.379 385477 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 2, update_id 15cf09e4-437c-4f1d-a624-fcd392d2e51d. Wait time elapsed: 0.004 2025-03-14 11:02:46.981 385477 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-14 11:02:46.879 385589 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-14 11:02:46.882 385589 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-14 11:02:46.883 385589 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none 2025-03-14 11:02:46.883 385589 INFO oslo.privsep.daemon [-] privsep daemon running as pid 385589 2025-03-14 11:02:46.984 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:46.984 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:46.984 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:48.083 385477 INFO neutron.agent.linux.interface [-] Device qr-8e2c27c0-75 already exists 2025-03-14 11:02:48.084 385477 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-dir', '/etc/neutron/conf.d/neutron-l3-agent', '--privsep_context', 'neutron.privileged.link_cmd', '--privsep_sock_path', '/tmp/tmpfwqx1nz9/privsep.sock'] 2025-03-14 11:02:48.095 385477 INFO neutron.agent.linux.interface [-] Device qr-cd332bdb-a2 already exists 2025-03-14 11:02:48.095 385477 INFO neutron.agent.linux.interface [-] Device qr-506ac36d-4e already exists 2025-03-14 11:02:48.104 385477 INFO neutron.agent.linux.interface [-] Device qr-5be154bc-e7 already exists 2025-03-14 11:02:48.674 385477 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap 2025-03-14 11:02:48.579 385671 INFO oslo.privsep.daemon [-] privsep daemon starting 2025-03-14 11:02:48.582 385671 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2025-03-14 11:02:48.584 385671 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2025-03-14 11:02:48.584 385671 INFO oslo.privsep.daemon [-] privsep daemon running as pid 385671 2025-03-14 11:02:48.677 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:48.677 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:48.677 385477 WARNING oslo_privsep.priv_context [-] privsep daemon already running 2025-03-14 11:02:50.020 385477 INFO neutron.agent.linux.interface [-] Device qr-f269cf3d-8a already exists 2025-03-14 11:02:50.294 385477 INFO neutron.agent.linux.interface [-] Device qr-8139dabf-34 already exists 2025-03-14 11:02:51.617 385477 INFO neutron.agent.linux.interface [-] Device qr-a4de62c7-b7 already exists 2025-03-14 11:02:51.641 385477 INFO neutron.agent.linux.interface [-] Device fg-0a5d1e67-87 already exists 2025-03-14 11:02:51.732 385477 INFO neutron.agent.linux.interface [-] Device qr-84c36ac1-9b already exists 2025-03-14 11:02:53.182 385477 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 15cf09e4-437c-4f1d-a624-fcd392d2e51d. Time elapsed: 6.802 2025-03-14 11:02:53.327 385477 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 74004c09-a8ad-43f1-a30b-6a199148477c. Time elapsed: 6.948 2025-03-14 11:02:54.040 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 5f780d66-8862-4d47-bd71-770d0682b569. Time elapsed: 7.662 2025-03-14 11:02:54.304 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 19e80549-a594-4f21-848b-1adb6636a2f3. Time elapsed: 7.929 2025-03-14 11:03:35.893 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 007111cb-6bd5-48c0-8522-6fefd60a02e5. Wait time elapsed: 0.001 2025-03-14 11:03:35.893 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 1, priority 1, update_id 007111cb-6bd5-48c0-8522-6fefd60a02e5. Wait time elapsed: 0.001 2025-03-14 11:03:35.894 385477 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 112ffbd5-d2b3-494a-a1e6-2d66d74d7951. Wait time elapsed: 0.002 2025-03-14 11:03:35.894 385477 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 1, priority 1, update_id 112ffbd5-d2b3-494a-a1e6-2d66d74d7951. Wait time elapsed: 0.002 2025-03-14 11:03:35.895 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 4abd01a3-a379-4f20-96c4-739757c799ca. Wait time elapsed: 0.001 2025-03-14 11:03:35.896 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 1, priority 1, update_id 4abd01a3-a379-4f20-96c4-739757c799ca. Wait time elapsed: 0.001 2025-03-14 11:03:35.896 385477 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id f56de0bd-04e6-4a0e-9428-2fa70fa62752. Wait time elapsed: 0.000 2025-03-14 11:03:35.896 385477 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 1, priority 1, update_id f56de0bd-04e6-4a0e-9428-2fa70fa62752. Wait time elapsed: 0.000 2025-03-14 11:03:36.631 385477 INFO neutron.agent.l3.agent [-] Finished a router delete for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 112ffbd5-d2b3-494a-a1e6-2d66d74d7951. Time elapsed: 0.737 2025-03-14 11:03:36.867 385477 INFO neutron.agent.l3.agent [-] Finished a router delete for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 4abd01a3-a379-4f20-96c4-739757c799ca. Time elapsed: 0.971 2025-03-14 11:03:36.935 385477 INFO neutron.agent.l3.agent [-] Finished a router delete for 668e4a57-9041-407f-b723-a64936d4be42, update_id 007111cb-6bd5-48c0-8522-6fefd60a02e5. Time elapsed: 1.042 2025-03-14 11:03:36.950 385477 INFO neutron.agent.l3.agent [-] Finished a router delete for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id f56de0bd-04e6-4a0e-9428-2fa70fa62752. Time elapsed: 1.053 2025-03-14 11:04:01.783 385477 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 7b48456f-a308-4ec4-b7cb-398cf1abded2. Wait time elapsed: 0.000 2025-03-14 11:04:01.783 385477 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 3, priority 1, update_id 7b48456f-a308-4ec4-b7cb-398cf1abded2. Wait time elapsed: 0.001 2025-03-14 11:04:01.784 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 05afa6d4-662b-4c09-9ab5-189491ded9ab. Wait time elapsed: 0.001 2025-03-14 11:04:01.784 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 05afa6d4-662b-4c09-9ab5-189491ded9ab. Wait time elapsed: 0.002 2025-03-14 11:04:01.785 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 8c84b780-2fb0-49af-9526-5fe0b740fab7. Wait time elapsed: 0.003 2025-03-14 11:04:01.786 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 8c84b780-2fb0-49af-9526-5fe0b740fab7. Wait time elapsed: 0.003 2025-03-14 11:04:01.786 385477 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d7f39fc6-e5db-489e-b129-dd078ee6f3e4. Wait time elapsed: 0.004 2025-03-14 11:04:01.787 385477 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id d7f39fc6-e5db-489e-b129-dd078ee6f3e4. Wait time elapsed: 0.004 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '668e4a57-9041-407f-b723-a64936d4be42': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent raise result 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017' for key 'PRIMARY'") 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent [parameters: {'port_id': '506ac36d-4e5c-47a1-96ca-36f739018dfe', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'506ac36d-4e5c-47a1-96ca-36f739018dfe-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'506ac36d-4e5c-47a1-96ca-36f739018dfe\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 11:04:02.362 385477 ERROR neutron.agent.l3.agent 2025-03-14 11:04:02.364 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 6e927d2d-bb82-4926-886b-2ef398d8b327. Wait time elapsed: 0.294 2025-03-14 11:04:02.364 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 3, priority 1, update_id 6e927d2d-bb82-4926-886b-2ef398d8b327. Wait time elapsed: 0.294 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for 'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3': oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 779, in _process_router_update 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(self.context, 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_utils/timeutils.py", line 383, in wrapper 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent result = func(*args, **kwargs) 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 938, in wrapper 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent ret = f(*args, **kwargs) 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 131, in get_routers 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent return cctxt.call(context, 'sync_routers', host=self.host, 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/neutron_lib/rpc.py", line 157, in call 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent result = self.transport._send( 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent return self._driver.send(target, ctxt, message, 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent raise result 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent oslo_messaging.rpc.client.RemoteError: Remote error: PendingRollbackError This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry '8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017' for key 'PRIMARY'") 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent [SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)] 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent [parameters: {'port_id': '8139dabf-344c-49c3-ae58-e83b2db9439a', 'host': 'hv-017', 'router_id': '4b89508c-beaf-43cf-92a6-9e080c03554d', 'vif_type': 'unbound', 'vif_details': '', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}] 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent (Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a) 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent ['Traceback (most recent call last):\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'pymysql.err.IntegrityError: (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n', '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in ensure_distributed_port_binding\n return record\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope\n yield resource\n', ' File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__\n next(self.gen)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 676, in _session\n self.session.flush()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj\n _emit_insert_statements(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements\n c = connection._execute_20(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute\n result = self._query(query)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query\n conn.query(q)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in _read_query_result\n result.read()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 131, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 178, in _ensure_host_set_on_ports\n self._ensure_host_set_on_port(\n', ' File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 253, in _ensure_host_set_on_port\n self.plugin.update_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped\n return method(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in update_distributed_port_binding\n binding = db.ensure_distributed_port_binding(\n', ' File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in ensure_distributed_port_binding\n return (context.session.query(models.DistributedPortBinding).\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one\n return self._iter().one()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n result = self.session.execute(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in execute\n conn = self._connection_for_bind(bind, close_with_result=True)\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind\n return self._transaction._connection_for_bind(\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind\n self._assert_active()\n', ' File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active\n raise sa_exc.PendingRollbackError(\n', 'sqlalchemy.exc.PendingRollbackError: This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry \'8139dabf-344c-49c3-ae58-e83b2db9439a-hv-017\' for key \'PRIMARY\'")\n[SQL: INSERT INTO ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]\n[parameters: {\'port_id\': \'8139dabf-344c-49c3-ae58-e83b2db9439a\', \'host\': \'hv-017\', \'router_id\': \'4b89508c-beaf-43cf-92a6-9e080c03554d\', \'vif_type\': \'unbound\', \'vif_details\': \'\', \'vnic_type\': \'normal\', \'profile\': \'\', \'status\': \'DOWN\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)\n']. 2025-03-14 11:04:02.482 385477 ERROR neutron.agent.l3.agent 2025-03-14 11:04:02.482 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 39000281-8669-44d7-9633-8af151979011. Wait time elapsed: 0.412 2025-03-14 11:04:02.482 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id 39000281-8669-44d7-9633-8af151979011. Wait time elapsed: 0.412 2025-03-14 11:04:03.728 385477 WARNING neutron.agent.l3.dvr_local_router [req-ba6b9a39-1471-496a-b639-3f9727c1461d 899cc34316484dfa8810461de99eb989 7221005084e14dcd937d5b0796d52ab0 - - -] Device qr-8e2c27c0-75 does not exist so ARP entry cannot be updated, will cache information to be applied later when the device exists 2025-03-14 11:04:06.778 385477 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=388263 2025-03-14 11:04:06.826 385477 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id d7f39fc6-e5db-489e-b129-dd078ee6f3e4. Time elapsed: 5.039 2025-03-14 11:04:06.826 385477 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ed25f570-d7d3-4b0f-a1bf-a870f0853d07. Wait time elapsed: 4.756 2025-03-14 11:04:06.826 385477 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 3, priority 1, update_id ed25f570-d7d3-4b0f-a1bf-a870f0853d07. Wait time elapsed: 4.757 2025-03-14 11:04:07.867 385477 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 39000281-8669-44d7-9633-8af151979011. Time elapsed: 5.384 2025-03-14 11:04:07.867 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id d5b02ad6-bf5d-4197-8217-2153b94db226. Wait time elapsed: 4.359 2025-03-14 11:04:07.868 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 3, priority 1, update_id d5b02ad6-bf5d-4197-8217-2153b94db226. Wait time elapsed: 4.360 2025-03-14 11:04:08.014 385477 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id ed25f570-d7d3-4b0f-a1bf-a870f0853d07. Time elapsed: 1.187 2025-03-14 11:04:08.978 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 6e927d2d-bb82-4926-886b-2ef398d8b327. Time elapsed: 6.614 2025-03-14 11:04:08.978 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 7aeb701b-f324-4115-84dd-5364c0b76c6c. Wait time elapsed: 6.154 2025-03-14 11:04:08.978 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 7aeb701b-f324-4115-84dd-5364c0b76c6c. Wait time elapsed: 6.155 2025-03-14 11:04:08.979 385477 INFO neutron.agent.l3.agent [-] Starting processing update ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 849c88fb-10c0-459d-80a2-a66ba67cce79. Wait time elapsed: 0.002 2025-03-14 11:04:08.980 385477 INFO neutron.agent.l3.agent [-] Starting router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, action 4, priority 0, update_id 849c88fb-10c0-459d-80a2-a66ba67cce79. Wait time elapsed: 0.002 2025-03-14 11:04:09.271 385477 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id d5b02ad6-bf5d-4197-8217-2153b94db226. Time elapsed: 1.404 2025-03-14 11:04:09.272 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 3d07e1ef-b0fe-4e40-98f7-1786746d0063. Wait time elapsed: 0.294 2025-03-14 11:04:09.272 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 3d07e1ef-b0fe-4e40-98f7-1786746d0063. Wait time elapsed: 0.295 2025-03-14 11:04:09.344 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 7b48456f-a308-4ec4-b7cb-398cf1abded2. Time elapsed: 7.561 2025-03-14 11:04:09.344 385477 INFO neutron.agent.l3.agent [-] Starting processing update 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 99f07407-68e7-42e7-a026-932baaeee4a5. Wait time elapsed: 5.862 2025-03-14 11:04:09.344 385477 INFO neutron.agent.l3.agent [-] Starting router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, action 4, priority 0, update_id 99f07407-68e7-42e7-a026-932baaeee4a5. Wait time elapsed: 5.862 2025-03-14 11:04:10.293 385477 INFO neutron.agent.l3.agent [-] Finished a router update for ebd75b85-e4e9-4c99-9966-17a1ea1a3b14, update_id 849c88fb-10c0-459d-80a2-a66ba67cce79. Time elapsed: 1.313 2025-03-14 11:04:10.508 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 7aeb701b-f324-4115-84dd-5364c0b76c6c. Time elapsed: 1.530 2025-03-14 11:04:10.508 385477 INFO neutron.agent.l3.agent [-] Starting processing update 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 59ebf23f-77b2-4d23-ad6a-41832b637d67. Wait time elapsed: 1.164 2025-03-14 11:04:10.508 385477 INFO neutron.agent.l3.agent [-] Starting router update for 668e4a57-9041-407f-b723-a64936d4be42, action 4, priority 0, update_id 59ebf23f-77b2-4d23-ad6a-41832b637d67. Wait time elapsed: 1.165 2025-03-14 11:04:10.756 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 4b89508c-beaf-43cf-92a6-9e080c03554d, update_id 99f07407-68e7-42e7-a026-932baaeee4a5. Time elapsed: 1.411 2025-03-14 11:04:10.840 385477 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 3d07e1ef-b0fe-4e40-98f7-1786746d0063. Time elapsed: 1.568 2025-03-14 11:04:10.840 385477 INFO neutron.agent.l3.agent [-] Starting processing update e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 965ee948-839c-4759-aca1-af4c31930f86. Wait time elapsed: 1.496 2025-03-14 11:04:10.840 385477 INFO neutron.agent.l3.agent [-] Starting router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, action 4, priority 0, update_id 965ee948-839c-4759-aca1-af4c31930f86. Wait time elapsed: 1.496 2025-03-14 11:04:11.778 385477 INFO neutron.agent.l3.agent [-] Finished a router update for 668e4a57-9041-407f-b723-a64936d4be42, update_id 59ebf23f-77b2-4d23-ad6a-41832b637d67. Time elapsed: 1.269 2025-03-14 11:04:12.157 385477 INFO neutron.agent.l3.agent [-] Finished a router update for e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3, update_id 965ee948-839c-4759-aca1-af4c31930f86. Time elapsed: 1.317