Open Stack

Hello everyone,

We have been investigating the cause behind the Jenkins Check 
gate-tempest-dsvm-networking-ovn failures (non-voting at the moment). The 
failures have been happening pretty consistently with every commit. I 
wanted to start a conversation to get some input as to why these errors 
may be happening.

One kind of error is related to the following (from the q-svc logs).

2015-08-04 05:40:28.313 ERROR neutron.agent.ovsdb.impl_idl 
[req-c189268a-1e1d-462f-a81e-62f0a34ff490 
tempest-FloatingIPAdminTestJSON-1706130555 
tempest-FloatingIPAdminTestJSON-1943105894] Traceback (most recent call 
last):
  File "/opt/stack/new/neutron/neutron/agent/ovsdb/native/connection.py", 
line 84, in run
    txn.results.put(txn.do_commit())
  File "/opt/stack/new/neutron/neutron/agent/ovsdb/impl_idl.py", line 99, 
in do_commit
    seqno)
  File "/opt/stack/new/neutron/neutron/agent/ovsdb/native/idlutils.py", 
line 125, in wait_for_change
    raise Exception("Timeout")
Exception: Timeout

When this error happens - in a separate thread there is DB Deadlock. Note 
that it's not always create_port (65%), it could be delete_port (30%), 
other calls (5%). There are many more of these errors (show below) than 
the above error.

But it is always: SQL: u'UPDATE ipavailabilityranges SET first_ip=%s WHERE 
ipavailabilityranges.allocation_pool_id = %s AND 
ipavailabilityranges.first_ip = %s AND ipavailabilityranges.last_ip = %s']

2015-08-04 05:39:37.303 9407 ERROR oslo_db.api   File 
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in 
wrapper
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api     return f(*args, 
**kwargs)
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api   File 
"/opt/stack/new/networking-ovn/networking_ovn/plugin.py", line 275, in 
create_port
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api     db_port = 
super(OVNPlugin, self).create_port(context, port)
...
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api   File 
"/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in 
execute
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api     self.errorhandler(self, 
exc, value)
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api   File 
"/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, 
in defaulterrorhandler
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api     raise errorclass, 
errorvalue
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api DBDeadlock: 
(_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; 
try restarting transaction') [SQL: u'UPDATE ipavailabilityranges SET 
first_ip=%s WHERE ipavailabilityranges.allocation_pool_id = %s AND 
ipavailabilityranges.first_ip = %s AND ipavailabilityranges.last_ip = %s'] 
[parameters: ('10.100.0.3', '851466c3-8d6b-4629-bf65-86be2f403e67', 
'10.100.0.2', '10.100.0.14')]
2015-08-04 05:39:37.303 9407 ERROR oslo_db.api 

Russell suggested removing the MYSQL_DRIVER=MySQL-python declaration from 
local.conf https://review.openstack.org/#/c/216413/ which results in 
PyMySQL as the default.

With the above change the above DB errors are no longer seen in my local 
setup, the CI setup is having trouble with the 
gate-networking-ovn-python27 test now therefore the 
gate-tempest-dsvm-networking-ovn never runs.

So there are 2 questions:

Is there any impact of using PyMySQL for the Jenkins check and gates.
Why is the gate-networking-ovn-python27 failing (the past couple of 
commits) in {0} 
networking_ovn.tests.unit.test_ovn_plugin.TestOvnPlugin.test_create_port_security 
[0.194020s] ... FAILED. Do we need another conversation to track this?

Amitabha

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150825/f98a4fb2/attachment.html>