Open Stack

Hi,

I'm happy to announce you that I just finished the last piece of the puzzle to add support for trollius coroutines in Oslo Messaging! See my two changes:

* Add a new aiogreen executor:
  https://review.openstack.org/#/c/136653/
* Add an optional executor callback to dispatcher:
  https://review.openstack.org/#/c/136652/

Related projects:

* asyncio is an event loop which is now part of Python 3.4:
  http://docs.python.org/dev/library/asyncio.html
* trollius is the port of the new asyncio module to Python 2:
  http://trollius.readthedocs.org/
* aiogreen implements the asyncio API on top of eventlet:
  http://aiogreen.readthedocs.org/

For the long story and the full history of my work on asyncio in OpenStack since one year, read:
http://aiogreen.readthedocs.org/openstack.html

The last piece of the puzzle is the new aiogreen project that I released a few days ago. aiogreen is well integrated and fully compatible with eventlet, it can be used in OpenStack without having to modify code. It is almost fully based on trollius, it just has a small glue to reuse eventlet event loop (get read/write notifications of file descriptors).

In the past, I tried to use the greenio project, which also implements the asyncio API, but it didn't fit well with eventlet. That's why I wrote a new project.

Supporting trollius coroutines in Oslo Messaging is just the first part of the global project. Here is my full plan to replace eventlet with asyncio.


First part (in progress): add support for trollius coroutines
-------------------------------------------------------------

Prepare OpenStack (Oslo Messaging) to support trollius coroutines using
``yield``: explicit asynchronous programming. Eventlet is still supported,
used by default, and applications and libraries don't need to be modified at
this point.

Already done:

* Write the trollius project: port asyncio to Python 2
* Stabilize trollius API
* Add trollius dependency to OpenStack
* Write the aiogreen project to provide the asyncio API on top of eventlet

To do:

* Stabilize aiogreen API
* Add aiogreen dependency to OpenStack
* Write an aiogreen executor for Oslo Messaging: rewrite greenio executor
  to replace greenio with aiogreen


Second part (to do): rewrite code as trollius coroutines
--------------------------------------------------------

Switch from implicit asynchronous programming (eventlet using greenthreads) to
explicit asynchronous programming (trollius coroutines using ``yield``). Need
to modify OpenStack Common Libraries and applications. Modifications can be
done step by step, the switch will take more than 6 months.

The first application candidate is Ceilometer. The Ceilometer project is young,
developers are aware of eventlet issues and like Python 3, and Ceilometer don't
rely so much on asynchronous programming: most time is spent into waiting the
database anyway.

The goal is to port Ceilometer to explicit asynchronous programming during the
cycle of OpenStack Kilo.

Some applications may continue to use implicit asynchronous programming. For
example, nova is probably the most complex part beacuse it is and old project
with a lot of legacy code, it has many drivers and the code base is large.

To do:

* Ceilometer: add trollius dependency and set the trollius event loop policy to
  aiogreen
* Ceilometer: change Oslo Messaging executor from "eventlet" to "aiogreen"
* Redesign the service class of Oslo Incubator to support aiogreen and/or
  trollius.  Currently, the class is designed for eventlet. The service class
  is instanciated before forking, which requires hacks on eventlet to update
  file descriptors.
* In Ceilometer and its OpenStack depedencencies: add new functions which
  are written with explicit asynchronous programming in mind (ex: trollius
  coroutines written with ``yield``).
* Rewrite Ceilometer endpoints (RPC methods) as trollius coroutines.

Questions:

* What about WSGI? aiohttp is not compatible with trollius yet.
* The quantity of code which need to be ported to asynchronous programming is
  unknown right now.
* We should be prepared to see deadlocks. OpenStack was designed for eventlet
  which implicitly switch on blocking operations. Critical sections may not be
  protected with locks, or not the right kind of lock.
* For performances, blocking operations can be executed in threads. OpenStack
  code is probably not thread-safe, which means new kinds of race conditions.
  But the code executed in threads will be explicitly scheduled to be executed
  in a thread (with ``loop.run_in_executor()``), so regressions can be easily
  identified.
* This part will take a lot of time. We may need to split it into subparts
  to have milestones, which is more attractive for developers.


Last part (to do): drop eventlet
--------------------------------

Replace aiogreen event loop with trollius event loop, drop aiogreen and drop
eventlet at the end.

This change will be done on applications one by one. This is no need to port
all applications at once. The work will start on Ceilometer, as a follow up
of the second part.

To do:

* Port remaining code to trollius
* Write a "trollius" executor for Oslo Messaging
* Ceilometer: Add a blocking call to ``loop.run_forever()`` in the ``main()``
  function
* Ceilometer: Replace "aiogreen" executor with "trollius" executor
* Ceilometer: Use the standard trollius event loop policy
* Ceilometer: drop the eventlet dependency

Questions:

* Putting a blocking call to ``loop.run_forever()`` may need to redesign
  Ceilometer, this part is unclear to me right now.

Optimization, can be done later:

* Oslo Messaging: watch directly the underlying file descriptor of sockets,
  instead of using a busy loop polling the notifier
* Ceilometer: use libraries supporting directly trollius to be able to run
  parallel tasks (ex: send multiple requests to a database)

Victor