<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 6/12/15 11:37 AM, Mike Bayer wrote:<br>
</div>
<blockquote cite="mid:557AFCC9.3070304@redhat.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 6/11/15 9:32 PM, Eugene Nikanorov
wrote:<br>
</div>
<blockquote
cite="mid:CAJfiwOTmAp4ohZ57o_F=pWJbEvHVmr4AYOS2QoVcUKSd8fgevw@mail.gmail.com"
type="cite">
<div dir="ltr">Hi neutrons,
<div><br>
</div>
<div>I'd like to draw your attention to an issue discovered by
rally gate job:</div>
<div><a moz-do-not-send="true"
href="http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE">http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE</a><br>
</div>
<div><br>
</div>
<div>I don't have bandwidth to take a deep look at it, but
first impression is that it is some issue with nested
transaction support either on sqlalchemy or pymysql side.</div>
<div>Also, besides errors with nested transactions, there are
a lot of Lock wait timeouts.</div>
<div><br>
</div>
<div>I think it makes sense to start with reverting the patch
that moves to pymysql.</div>
</div>
</blockquote>
My immediate reaction is that this is perhaps a
concurrency-related issue; because PyMySQL is pure python and
allows for full blown eventlet monkeypatching, I wonder if somehow
the same PyMySQL connection is being used in multiple contexts.
E.g. one greenlet starts up a savepoint, using identifier "_3"
which is based on a counter that is local to the SQLAlchemy
Connection, but then another greenlet shares that PyMySQL
connection somehow with another SQLAlchemy Connection that uses
the same identifier.<br>
</blockquote>
<br>
reading more of the log, it seems the main issue is just that
there's a deadlock on inserting into the securitygroups table. The
deadlock on insert can be because of an index being locked.<br>
<br>
<br>
I'd be curious to know how many greenlets are running concurrently
here, and what the overall transaction looks like within the
operation that is failing here (e.g. does each transaction insert
multiple rows into securitygroups? that would make a deadlock seem
more likely).<br>
<br>
</body>
</html>