Open Stack

On Tue, Jun 16, 2015 at 12:33 AM, Kevin Benton <blak111 at gmail.com> wrote:
>>Do these kinds of test even make sense? And are they feasible at all? I
>> doubt we have any framework for injecting anything in neutron code under
>> test.
>
> I was thinking about this in the context of a lot of the fixes we have for
> other concurrency issues with the database. There are several exception
> handlers that aren't exercised in normal functional, tempest, and API tests
> because they require a very specific order of events between workers.
>
> I wonder if we could write a small shim DB driver that wraps the python one
> for use in tests that just makes a desired set of queries take a long time
> or fail in particular ways? That wouldn't require changes to the neutron
> code, but it might not give us the right granularity of control.

Might be worth a look.

>>Finally, please note I am using DB-level locks rather than non-locking
>> algorithms for making reservations.
>
> I thought these were effectively broken in Galera clusters. Is that not
> correct?

As I understand it, if two writes to two different masters end up
violating some db-level constraint then the operation will cause a
failure regardless if there is a lock.

Basically, on Galera, instead of waiting for the lock, each will
proceed with the transaction.  Finally, on commit, a write
certification will double check constraints with the rest of the
cluster (with a write certification).  It is at this point where
Galera will fail one of them as a deadlock for violating the
constraint.  Hence the need to retry.  To me, non-locking just means
that you embrace the fact that the lock won't work and you don't
bother to apply it in the first place.

If my understanding is incorrect, please set me straight.

> If you do go that route, I think you will have to contend with DBDeadlock
> errors when we switch to the new SQL driver anyway. From what I've observed,
> it seems that if someone is holding a lock on a table and you try to grab
> it, pymsql immediately throws a deadlock exception.

I'm not familiar with pymysql to know if this is true or not.  But,
I'm sure that it is possible not to detect the lock at all on galera.
Someone else will have to chime in to set me straight on the details.

Carl