Open Stack

Hey

This is an attempt to summarize a really useful discussion that Victor,
Flavio and I have been having today. At the bottom are some background
links - basically what I have open in my browser right now thinking
through all of this.

We're attempting to take baby-steps towards moving completely from
eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
first victim.

Ceilometer's code is run in response to various I/O events like REST API
requests, RPC calls, notifications received, etc. We eventually want the
asyncio event loop to be what schedules Ceilometer's code in response to
these events. Right now, it is eventlet doing that.

Now, because we're using eventlet, the code that is run in response to
these events looks like synchronous code that makes a bunch of
synchronous calls. For example, the code might do some_sync_op() and
that will cause a context switch to a different greenthread (within the
same native thread) where we might handle another I/O event (like a REST
API request) while we're waiting for some_sync_op() to return:

  def foo(self):
      result = some_sync_op()  # this may yield to another greenlet
      return do_stuff(result)

Eventlet's infamous monkey patching is what make this magic happen.

When we switch to asyncio's event loop, all of this code needs to be
ported to asyncio's explicitly asynchronous approach. We might do:

  @asyncio.coroutine
  def foo(self):
      result = yield from some_async_op(...)
      return do_stuff(result)

or:

  @asyncio.coroutine
  def foo(self):
      fut = Future()
      some_async_op(callback=fut.set_result)
      ...
      result = yield from fut
      return do_stuff(result)

Porting from eventlet's implicit async approach to asyncio's explicit
async API will be seriously time consuming and we need to be able to do
it piece-by-piece.

The question then becomes what do we need to do in order to port a
single oslo.messaging RPC endpoint method in Ceilometer to asyncio's
explicit async approach?

The plan is:

  - we stick with eventlet; everything gets monkey patched as normal

  - we register the greenio event loop with asyncio - this means that 
    e.g. when you schedule an asyncio coroutine, greenio runs it in a 
    greenlet using eventlet's event loop

  - oslo.messaging will need a new variant of eventlet executor which 
    knows how to dispatch an asyncio coroutine. For example:

        while True:
            incoming = self.listener.poll()
            method = dispatcher.get_endpoint_method(incoming)
            if asyncio.iscoroutinefunc(method):
                result = method()
                self._greenpool.spawn_n(incoming.reply, result)
            else:
                self._greenpool.spawn_n(method)

    it's important that even with a coroutine endpoint method, we send 
    the reply in a greenthread so that the dispatch greenthread doesn't
    get blocked if the incoming.reply() call causes a greenlet context
    switch

  - when all of ceilometer has been ported over to asyncio coroutines, 
    we can stop monkey patching, stop using greenio and switch to the 
    asyncio event loop

  - when we make this change, we'll want a completely native asyncio 
    oslo.messaging executor. Unless the oslo.messaging drivers support 
    asyncio themselves, that executor will probably need a separate
    native thread to poll for messages and send replies.

If you're confused, that's normal. We had to take several breaks to get
even this far because our brains kept getting fried.

HTH,
Mark.

Victor's excellent docs on asyncio and trollius:

  https://docs.python.org/3/library/asyncio.html
  http://trollius.readthedocs.org/

Victor's proposed asyncio executor:

  https://review.openstack.org/70948

The case for adopting asyncio in OpenStack:

  https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio

A previous email I wrote about an asyncio executor:

 http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html

The mock-up of an asyncio executor I wrote:

  https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py

My blog post on async I/O and Python:

  http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/

greenio - greelets support for asyncio:

  https://github.com/1st1/greenio/