Open Stack

On Tue, 12 May 2015 at 05:08 Mike Bayer <mbayer at redhat.com> wrote:
>
> On 5/11/15 2:02 PM, Attila Fazekas wrote:
> > The scary part of a blocking I/O call is when you have two
> > python thread (or green thread) and one of them is holding a DB lock the
> other
> > is waiting for the same lock in a native blocking I/O syscall.
> that's a database deadlock and whether you use eventlet, threads,
> asycnio or even just two transactions in a single-threaded script, that
> can happen regardless.  if your two eventlet "non blocking" greenlets
> are waiting forever for a deadlock,  you're just as deadlocked as if you
> have OS threads.
>

Not true (if I understand the situation Attila is referring to).

> If you do a read(2) in native code, the python itself might not be able
> to preempt it
> > Your transaction might be finished with `DB Lock wait timeout`,
> > with 30 sec of doing nothing, instead of scheduling to the another
> python thread,
> > which would be able to release the lock.
>
>
> Here's the "you're losing me" part because Python threads are OS
> threads, so Python isn't directly involved trying to "preempt" anything,
> unless you're referring to the effect of the GIL locking up the
> program.   However, it's pretty easy to make two threads in Python hit a
> database and do a deadlock against each other, and the rest of the
> program's threads continue to run just fine; in a DB deadlock situation
> you are blocked on IO and IO releases the GIL.
>
> If you can illustrate a test script that demonstrates the actual failing
> of OS threads that does not occur greenlets here, that would make it
> immediately apparent what it is you're getting at here.
>

1. Thread A does something that takes a lock on the DB side
2. Thread B does something that blocks waiting for that same DB lock
3. Depends on the threading model - see below

In a "true" preemptive threading system (eg: regular python threads), (3)
is:

3.  Eventually A finishes its transaction/whatever, commits and releases
the DB lock
4. B then takes the lock and proceeds
5. Profit

However, in a system where B's DB client can't be preempted (eg: eventlet
or asyncio calling into a C-based mysql library, and A and B are running on
the same underlying kernel thread), (3) is:

3. B will never be preempted, A will never be rescheduled, and thus A will
never complete whatever it was doing.
4. Deadlock (in mysql-python's case, until a deadlock timer raises an
exception and kills B 30s later)
5. Sadness.  More specifically, we add a @retry to paper over the
particular observed occurrence and then repeat this discussion on os-dev
when the topic comes up again 6 months later.

Note that this is not the usual database transaction deadlock caused by A
and B each taking a lock and then trying to take the other's lock - this is
a deadlock purely in the client-side code caused entirely by the lack of
preemption during an otherwise safe series of DB operations.

See my oslo.db unittest in Ib35c95defea8ace5b456af28801659f2ba67eb96 that
reproduces the above with eventlet and allows you to test the behaviour of
various DB drivers.

(zzzeek: I know you've already seen all of the above in previous
discussions, so sorry for repeating).

 - Gus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150512/b37580d0/attachment.html>