Open Stack

On 06/12/2015 06:31 AM, Joe Gordon wrote:
> 
> 
> On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     On 06/12/2015 01:17 AM, Salvatore Orlando wrote:
>     > It is however interesting that both "lock wait timeouts" and "missing
>     > savepoint" errors occur in operations pertaining the same table -
>     > securitygroups in this case.
>     > I wonder if the switch to pymysl has not actually uncovered some other
>     > bug in Neutron.
>     >
>     > I have no opposition to a revert, but since this will affect most
>     > projects, it's probably worth finding some time to investigate what is
>     > triggering this failure when sqlalchemy is backed by pymysql before
>     > doing that.
> 
>     Right, we knew that the db driver would move some bugs around because
>     we're no longer blocking python processes on db access (so there used to
>     be a pseudo synchronization point before you ever got to the database).
> 
>     My feeling is this should be looked into before it is straight reverted
>     (are jobs failing beyond Rally?). There are a number of benefits with
> 
> 
> A quick look at logstash.openstack.org <http://logstash.openstack.org>
> shows some of the stacktraces are happening in other neutron jobs as well.
>  
> 
>     the new driver, and we can't get to python3 with the old one.
> 
> 
> Agreed, pymysql is not to blame it looks like we have hit some neutron
> issues.  So lets try to fix neutron. Just because neutron reverts the
> default sql connector doesn't mean operators won't end up trying pymysql.
>  
> 
> 
>     Rally failing is also an indicator that just such and implicit lock was
>     behavior that was depended on before, because it will be sending a bunch
>     of similar operations all at once as a kind of stress test. It would
>     tend to expose issues like this first.
> 
> 
> Glad to see us catch these issues early.

So, looking at little deeper this is also exposing across the board as
well. I filed this bug - https://bugs.launchpad.net/neutron/+bug/1464612.

I'm going to trigger and push the revert, as it's blocked fixes for
stable/kilo being able to merge code (so it's cascading).

We can decide if we bring this back for all non-neutron jobs once things
are working again (it lives behind a config var in devstack, so easy to
set it per job).

	-Sean

-- 
Sean Dague
http://dague.net