Open Stack

On 6/12/15 6:42 AM, Sean Dague wrote:
> On 06/12/2015 06:31 AM, Joe Gordon wrote:
>>
>> On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague <sean at dague.net
>> <mailto:sean at dague.net>> wrote:
>>
>>      On 06/12/2015 01:17 AM, Salvatore Orlando wrote:
>>      > It is however interesting that both "lock wait timeouts" and "missing
>>      > savepoint" errors occur in operations pertaining the same table -
>>      > securitygroups in this case.
>>      > I wonder if the switch to pymysl has not actually uncovered some other
>>      > bug in Neutron.
>>      >
>>      > I have no opposition to a revert, but since this will affect most
>>      > projects, it's probably worth finding some time to investigate what is
>>      > triggering this failure when sqlalchemy is backed by pymysql before
>>      > doing that.
>>
>>      Right, we knew that the db driver would move some bugs around because
>>      we're no longer blocking python processes on db access (so there used to
>>      be a pseudo synchronization point before you ever got to the database).
>>
>>      My feeling is this should be looked into before it is straight reverted
>>      (are jobs failing beyond Rally?). There are a number of benefits with
>>
>>
>> A quick look at logstash.openstack.org <http://logstash.openstack.org>
>> shows some of the stacktraces are happening in other neutron jobs as well.
>>   
>>
>>      the new driver, and we can't get to python3 with the old one.
>>
>>
>> Agreed, pymysql is not to blame it looks like we have hit some neutron
>> issues.  So lets try to fix neutron. Just because neutron reverts the
>> default sql connector doesn't mean operators won't end up trying pymysql.
>>   
>>
>>
>>      Rally failing is also an indicator that just such and implicit lock was
>>      behavior that was depended on before, because it will be sending a bunch
>>      of similar operations all at once as a kind of stress test. It would
>>      tend to expose issues like this first.
>>
>>
>> Glad to see us catch these issues early.
> So, looking at little deeper this is also exposing across the board as
> well. I filed this bug - https://bugs.launchpad.net/neutron/+bug/1464612.
>
> I'm going to trigger and push the revert, as it's blocked fixes for
> stable/kilo being able to merge code (so it's cascading).
Please keep me very deeply in the loop on this because it will save 
everyone a lot of time if I can decipher the interaction with SQLAlchemy 
internals for folks.




>
> We can decide if we bring this back for all non-neutron jobs once things
> are working again (it lives behind a config var in devstack, so easy to
> set it per job).
>
> 	-Sean
>