Open Stack

Thanks for taking this on, Victor.

Do you have a small eventlet program example that you can show the
transformation to asyncio?

Thanks, -peter

On Tue, Feb 4, 2014 at 8:38 AM, victor stinner
<victor.stinner at enovance.com> wrote:
> Hi,
>
> I would like to replace eventlet with asyncio in OpenStack for the asynchronous programming. The new asyncio module has a better design and is less "magical". It is now part of python 3.4 arguably becoming the de-facto standard for asynchronous programming in Python world.
>
>
> The asyncio module is a new module of Python 3.4 written by Guido van Rossum as an abstraction of existing event loop (Twisted, Tornado, eventlet, etc.). In fact, it's more than an abstraction: it has its own event loop and can be used alone. For the background, read the PEP:
>
>    http://www.python.org/dev/peps/pep-3156/
>    "Asynchronous IO Support Rebooted: the "asyncio" Module"
>
> For more information on asyncio, see its documentation:
>
>    http://docs.python.org/dev/library/asyncio.html
>
> The asyncio module is also known as "Tulip" which is third-party project written for Python 3.3. Tulip was written first and then it was integrated in Python 3.4.
>
>    http://code.google.com/p/tulip/
>
> The main difference between eventlet and asyncio is that context switching between two concurrent tasks is explicit. When a task "blocks"  (ex: wait for an event), it should use the "yield from" syntax which will switch to the next task: it's similar to "greenlet.switch" in eventlet. So it becomes obvious which parts of the code do switch and which don't. With eventlet, you have to read the source code of a function before calling it to check if it may call greenlet.switch() or not, and so debugging is more difficult. Or worse, the function may switch in a new version of a module, you won't notice the change.
>
> The asyncio module handles various kind of events: sockets, pipes, subprocesses, UNIX signals, etc. All these things are handled in a single event loop. You can uses an "exector" to run a blocking task in a pool of thread. There is a "low-level" API using transports and protocols, similar to Twisted transports and protocols. But there is also a "high-level" API using streams, which gives a syntax close to eventlet (except that you have to add "yield from"). See an example to send an HTTP request and print received HTTP headers, the "yield from reader.readline()" instruction "blocks" until it gets a full line:
>
>    http://docs.python.org/dev/library/asyncio-stream.html#example
>
>
> The problem is that the asyncio module was written for Python 3.3, whereas OpenStack is not fully Python 3 compatible (yet). To easy the transition I have ported asyncio on Python 2, it's the new Trollis project which supports Python 2.6-3.4:
>
>    https://bitbucket.org/enovance/trollius
>
> The Trollius API is the same than asyncio, the main difference is the syntax in coroutines: "yield from task" must be written "yield task", and "return value" must be written "raise Return(value)".
>
>
>
> The first step to move from eventlet to asyncio is a new executor using Trollius in Olso Messaging:
>
>   https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio
>   https://review.openstack.org/#/c/70948
>
>
> The transition from eventlet to asyncio can be done step by step. Thanks to the greenio project, asyncio can reuse the greenlet event loop (and so run in the main thread). So asyncio and eventlet become "compatible". While asyncio can also run its own event loop in a separated thread.
>
> If eventlet is completely replaced with asyncio in a project, greenio can be dropped, and asyncio event loop can be run its own event loop in the main thread.
>
> When OpenStack will be compatible with Python 3.3, it will be possible to use the builtin asyncio module of Python 3.4 directly instead of Trollus. Since "yield from" is incompatible with Python 2, some parts of the code may need to have two versions (one for Python 2, one for Python 3) if we want to use the "Python 3 flavor" of asyncio... or Python 2 support might be simplify dropped.
>
> Victor
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev