Open Stack

----- Original Message -----
> From: "Mike Bayer" <mbayer at redhat.com>
> To: openstack-dev at lists.openstack.org
> Sent: Monday, May 11, 2015 9:07:13 PM
> Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
> 
> 
> 
> On 5/11/15 2:02 PM, Attila Fazekas wrote:
> >
> > Not just with local database connections,
> > the 10G network itself also fast. Is is possible you spend more time even
> > on
> > the kernel side tcp/ip stack (and the context switch..) (Not in physical
> > I/O wait)
> > than in the actual work on the DB side. (Check netperf TCP_RR)
> >
> > The scary part of a blocking I/O call is when you have two
> > python thread (or green thread) and one of them is holding a DB lock the
> > other
> > is waiting for the same lock in a native blocking I/O syscall.
> that's a database deadlock and whether you use eventlet, threads,
> asycnio or even just two transactions in a single-threaded script, that
> can happen regardless.  if your two eventlet "non blocking" greenlets
> are waiting forever for a deadlock,  you're just as deadlocked as if you
> have OS threads.
> 
> 
> > If you do a read(2) in native code, the python itself might not be able to
> > preempt it
> > Your transaction might be finished with `DB Lock wait timeout`,
> > with 30 sec of doing nothing, instead of scheduling to the another python
> > thread,
> > which would be able to release the lock.
> 
> 
> Here's the "you're losing me" part because Python threads are OS
> threads, so Python isn't directly involved trying to "preempt" anything,
> unless you're referring to the effect of the GIL locking up the
> program.   However, it's pretty easy to make two threads in Python hit a
> database and do a deadlock against each other, and the rest of the
> program's threads continue to run just fine; in a DB deadlock situation
> you are blocked on IO and IO releases the GIL.
> 
> If you can illustrate a test script that demonstrates the actual failing
> of OS threads that does not occur greenlets here, that would make it
> immediately apparent what it is you're getting at here.
>

http://www.fpaste.org/220824/raw/

I just put together hello word C example and a hello word threading example,
and replaced the print with sleep(3).

When I use the sleep(3) from python, the 5 thread program runs in ~3 second,
when I use the sleep(3) from native code, it runs ~15 sec.

So yes, it is very likely a GIL lock wait related issue,
when the native code is not assisting.
 
Do you need a DB example, by using the mysql C driver,
and waiting in an actual I/O primitive ?

The greenthreads will not help here.

If I would import the python time.sleep from the C code it might help.

Using pure python driver helps to avoid this kind of issues,
but in this case you have the `cPython is slow` issue.

> 
> >
> >> [1]
> >> http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>