Open Stack

On 7/4/14, 4:45 AM, Julien Danjou wrote:
> On Thu, Jul 03 2014, Mark McLoughlin wrote:
>
>> We're attempting to take baby-steps towards moving completely from
>> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
>> first victim.
> Thumbs up for the plan, that sounds like a good approach from what I
> got. I just think there's a lot of things that are going to be
> synchronous anyway because not everything provide a asynchronous
> alternative (i.e. SQLAlchemy or requests don't yet AFAIK). It doesn't
> worry me much as there nothing we can do on our side, except encourage
> people to stop writing synchronous API¹.
>
> And big +1 for using Ceilometer as a test bed. :)
Allowing SQLAlchemy to be fully compatible with an explicit async
programming approach, which note is distinctly different from allowing
SQLAlchemy to run efficiently within an application that uses explicit
async, has been studied and as of yet, it does not seem possible without
ruining the performance of the library (including Core-only),
invalidating the ORM entirely, and of course doing a rewrite of almost
the whole thing (see
http://stackoverflow.com/questions/16491564/how-to-make-sqlalchemy-in-tornado-to-be-async/16503103#16503103,
http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html#gevent-and-pep-3156).   


But before you even look at database abstraction layers, you need a
database driver.  What's the explicitly async-compatible driver for
MySQL?    Googling around I found
https://github.com/eliast/async-MySQL-python, but not much else.  Note
that for explicit async, a driver that allows monkeypatching is no
longer enough.  You need an API like Psycopg2s asynchronous support:
http://initd.org/psycopg/docs/advanced.html#async-support.  Note that
psycopg2's API is entirely an extension to the Python DBAPI:
http://legacy.python.org/dev/peps/pep-0249/.  So an all explicit async
approach necessitates throwing out this out as well; as an alternative,
here is twisted's adbapi extension to pep-249's API:
https://twistedmatrix.com/documents/current/core/howto/rdbms.html.   I'm
not sure if Twisted provides an explicit async API for MySQL.

If you are writing an application that runs in an explicit, or even an
implicitly async system, and your database driver isn't compatible with
that, your application will perform terribly - because you've given up
regular old threads, and your app now serializes most of what it does
through a single, blocking pipe.    That is the current status of all
Openstack apps that rely heavily on MySQLdb and Eventlet at the same
time.   Explicitly asyncing it will help in that we won't get
hard-to-predict context switches that deadlock against the DB driver
(also solvable just by using an appropriate patchable driver), but it
won't help performance until that is solved.

Nick's post points the way towards a way that everyone can have what
they want - which is that once we get a MySQL database adapter that is
implicit-async-patch-capable, the explicit async parts of openstack call
into database routines that are using implicit async via a gevent-like
approach.   That way SQLAlchemy's source code does not have to multiply
it's function call count by an order of magnitude, or be rewritten, and
the ORM-like features that folks like to complain about as they continue
to use them like crazy (e.g. lazy loading) can remain intact.

If we are in fact considering going down this latest rabbit hole which
claims that program code cannot possibly be efficient or trusted unless
all blocking operations are entirely written literally by humans,
yielding all the way down to the system calls, I would ask that we make
a concerted effort to face just exactly what that means we'd be giving
up.   Because the cost of how much has to be thrown away may be
considerably higher than people might realize.    For those parts of an
app that make sense using explicit async, we should be doing so. 
However, we should ensure that those code sections more appropriate as
implicit async remain first class citizens as well.