Open Stack

Hi, all.
I hit an error about ovs agents state report after deployed openstack by chef. I check the code in agents_db. Found there are some greenthread.sleep(0) in _create_or_update_agent ( https://github.com/openstack/neutron/commit/21bf6f7e4945bc7e8c303273ad10c28b6cfc8b08 ). I'm not sure why we need those sleep here. But I think this can let the user hit DBDuplicateEntry excetpion(there already have an retry to avoid this error, but it still not safe, see https://github.com/openstack/neutron/blob/ebc3ccfdd070fb69274069057c7c1cd8f37b508a/neutron/db/agents_db.py#L198 ). This error is not easy to reproduce. I think if we make several rpc calls for create_or_update_agent at the same time before this agents has already been stored in db, we can encounter this error again.

Remove those sleep can avoid this issue, but seems those sleep is used to fix some bugs. Need someone's help to fix this issue.

2014-11-10 23:14:36.050 31989 ERROR oslo.messaging.rpc.dispatcher [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Exception during message handling: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}') 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in report_state 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.plugin.create_or_update_agent(context, agent_state) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in create_or_update_agent 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__ 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in create_or_update_agent 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 191, in _create_or_update_agent 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher greenthread.sleep(0) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__ 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.rollback() 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__ 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__ 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.commit() 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in commit 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self._prepare_impl() 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.session.flush() 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in flush 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self._flush(objects) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in _flush 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher transaction.rollback(_capture_exception=True) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__ 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in _flush 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher flush_context.execute() 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher rec.execute(self) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher uow 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher table, insert) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher execute(statement, params) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher params) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compiled_sql, distilled_params 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher context) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher six.reraise(type(newraise), newraise, sys.exc_info()[2]) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher per_fn = fn(ctx) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher context.is_disconnect) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223, in _db2_dupe_key_error 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher raise exception.DBDuplicateEntry([], integrity_error) 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher DBDuplicateEntry: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}') 
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher 
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Returning exception (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}') to caller 
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common [req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in report_state\n self.plugin.create_or_update_agent(context, agent_state)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in create_or_update_agent\n return self._create_or_update_agent(context, agent)\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in create_or_update_agent\n return self._create_or_update_agent(context, agent)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 191, in _create_or_update_agent\n greenthread.sleep(0)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\n self.rollback()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\n self.commit()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in commit\n self._prepare_impl()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl\n self.session.flush()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute\n uow\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj\n table, insert)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569, in _emit_insert_statements\n execute(statement, params)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute\n params)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement\n compiled_sql, distilled_params\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context\n context)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception\n six.reraise(type(newraise), newraise, sys.exc_info()[2])\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception\n per_fn = fn(ctx)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler\n context.is_disconnect)\n', ' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223, in _db2_dupe_key_error\n raise exception.DBDuplicateEntry([], integrity_error)\n', 'DBDuplicateEntry: (IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 \'INSERT INTO agents (id, agent_type, "binary", topic, host, admin_state_up, created_at, started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\' (\'5028b520-1a90-46c2-9e97-b1cb80d6ebe0\', \'Open vSwitch agent\', \'neutron-openvswitch-agent\', \'N/A\', \'testhl12.sce.ibm.com\', \'1\', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), None, \'{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false, "bridge_mappings": {"default": "br-eth1"}}\')\n']




huwei.xtu at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141113/a90d78ad/attachment.html>