Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/07/14 05:30, Mark McLoughlin wrote:
> Hey
> 
> This is an attempt to summarize a really useful discussion that Victor,
> Flavio and I have been having today. At the bottom are some background
> links - basically what I have open in my browser right now thinking
> through all of this.
> 
> We're attempting to take baby-steps towards moving completely from
> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
> first victim.

Has this been widely agreed on? It seems to me like we are mixing two
issues:
1) we need to move to py3
2) some people want to move from eventlet (I am not convinced that the
   volume of code changes warrants the end goal - and review load)

To achieve "1)" in a lower risk change, shouldn't we rather run eventlet
on top of asyncio? - i.e. not require widespread code changes.

So we can maintain the main loop API but move to py3. I am not sure on
the feasibility, but seems to me like a more contained change.

- -Angus

> 
> Ceilometer's code is run in response to various I/O events like REST API
> requests, RPC calls, notifications received, etc. We eventually want the
> asyncio event loop to be what schedules Ceilometer's code in response to
> these events. Right now, it is eventlet doing that.
> 
> Now, because we're using eventlet, the code that is run in response to
> these events looks like synchronous code that makes a bunch of
> synchronous calls. For example, the code might do some_sync_op() and
> that will cause a context switch to a different greenthread (within the
> same native thread) where we might handle another I/O event (like a REST
> API request) while we're waiting for some_sync_op() to return:
> 
>   def foo(self):
>       result = some_sync_op()  # this may yield to another greenlet
>       return do_stuff(result)
> 
> Eventlet's infamous monkey patching is what make this magic happen.
> 
> When we switch to asyncio's event loop, all of this code needs to be
> ported to asyncio's explicitly asynchronous approach. We might do:
> 
>   @asyncio.coroutine
>   def foo(self):
>       result = yield from some_async_op(...)
>       return do_stuff(result)
> 
> or:
> 
>   @asyncio.coroutine
>   def foo(self):
>       fut = Future()
>       some_async_op(callback=fut.set_result)
>       ...
>       result = yield from fut
>       return do_stuff(result)
> 
> Porting from eventlet's implicit async approach to asyncio's explicit
> async API will be seriously time consuming and we need to be able to do
> it piece-by-piece.
> 
> The question then becomes what do we need to do in order to port a
> single oslo.messaging RPC endpoint method in Ceilometer to asyncio's
> explicit async approach?
> 
> The plan is:
> 
>   - we stick with eventlet; everything gets monkey patched as normal
> 
>   - we register the greenio event loop with asyncio - this means that 
>     e.g. when you schedule an asyncio coroutine, greenio runs it in a 
>     greenlet using eventlet's event loop
> 
>   - oslo.messaging will need a new variant of eventlet executor which 
>     knows how to dispatch an asyncio coroutine. For example:
> 
>         while True:
>             incoming = self.listener.poll()
>             method = dispatcher.get_endpoint_method(incoming)
>             if asyncio.iscoroutinefunc(method):
>                 result = method()
>                 self._greenpool.spawn_n(incoming.reply, result)
>             else:
>                 self._greenpool.spawn_n(method)
> 
>     it's important that even with a coroutine endpoint method, we send 
>     the reply in a greenthread so that the dispatch greenthread doesn't
>     get blocked if the incoming.reply() call causes a greenlet context
>     switch
> 
>   - when all of ceilometer has been ported over to asyncio coroutines, 
>     we can stop monkey patching, stop using greenio and switch to the 
>     asyncio event loop
> 
>   - when we make this change, we'll want a completely native asyncio 
>     oslo.messaging executor. Unless the oslo.messaging drivers support 
>     asyncio themselves, that executor will probably need a separate
>     native thread to poll for messages and send replies.
> 
> If you're confused, that's normal. We had to take several breaks to get
> even this far because our brains kept getting fried.
> 
> HTH,
> Mark.
> 
> Victor's excellent docs on asyncio and trollius:
> 
>   https://docs.python.org/3/library/asyncio.html
>   http://trollius.readthedocs.org/
> 
> Victor's proposed asyncio executor:
> 
>   https://review.openstack.org/70948
> 
> The case for adopting asyncio in OpenStack:
> 
>   https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio
> 
> A previous email I wrote about an asyncio executor:
> 
>  http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html
> 
> The mock-up of an asyncio executor I wrote:
> 
>   https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py
> 
> My blog post on async I/O and Python:
> 
>   http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/
> 
> greenio - greelets support for asyncio:
> 
>   https://github.com/1st1/greenio/
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTuuLsAAoJEFrDYBLxZjWotFwH/1PYbrXFYb+gYNCWBAtGJ3mi
wHdHIonIjcD3Zpnvs3o5BJDbugSlfdKzHRjSy9dQG5LyuihiFNK3AFQu/ldfJEdS
/QiLIaCQYI/13AwWA9+8A/TYpoyVJytVaXm3uBp2fMhy02TYWOweh02gifUZ7CBp
3hRzGQE4EmHr0YpgQRupIDpE50GzflB8fySXyfm3Cme7vcCivyMOHexsxKe3Bm2o
LxfMRPvUOqZ9MaJ6XFTy8SieNA4kMdZYmrm4pu5nrALXwxZ1dAtpD4jcjLK7Jh5T
P+eZHcQTe7bYSlk+LMchfHpAlTuLTySHv/LB6wvt2ljSM4AAmABKOKwAjFFL66k=
=O09e
-----END PGP SIGNATURE-----