Open Stack

Hi,

Le lundi 7 juillet 2014, 12:48:59 Nikola Đipanov a écrit :
> When I read all of this stuff and got my head around it (took some time
> :) ), a glaring drawback of such an approach, and as I mentioned on the
> spec proposing it [1] is that we would not really doing asyncio, we
> would just be pretending we are by using a subset of it's APIs, and
> having all of the really important stuff for overall design of the code
> (code that needs to do IO in the callbacks for example) and ultimately -
> performance, completely unavailable to us when porting.

The global plan is to:

1. use asyncio API
2. detect code relying on implicit scheduling and patch it to use explicit 
scheduling (use the coroutine syntax with yield)
3. "just" change the event loop from greenio to a classic "select" event loop 
(select, poll, epoll, kqueue, etc.) of Trollius

I see asyncio as an API: it doesn't really matter which event loop is used, 
but I want to get rid of eventlet :-)

> So in Mark's example above:
> 
>   @asyncio.coroutine
>   def foo(self):
>     result = yield from some_async_op(...)
>     return do_stuff(result)
> 
> A developer would not need to do anything that asyncio requires like
> make sure that some_async_op() registers a callback with the eventloop
> (...)

It's not possible to break the world right now, some people will complain :-)

The idea is to have a smooth transition. We will write tools to detect 
implicit scheduling and fix code. I don't know the best option for that right 
now (monkey-patch eventlet, greenio or trollius?).

> So I hacked up together a small POC of a different approach. In short -
> we actually use a real asyncio selector eventloop in a separate thread,
> and dispatch stuff to it when we figure out that our callback is in fact
> a coroutine.

See my previous attempty: the asyncio executor runs the asyncio event loop in 
a dedicated thread:
https://review.openstack.org/#/c/70948/

I'm not sure that it's possible to use it in OpenStack right now because the 
whole Python standard library is monkey patched, including the threading 
module.

The issue is also to switch the control flow between the event loop thread and 
the main thread. There is no explicit event loop in the main thread. The most 
obvious solution for that is to schedule tasks using eventlet...

That's exactly the purpose of greenio: glue between asyncio and greenlet. And 
using greenio, there is no need of running a new event loop in a thread, which 
makes the code simpler.

> (..) we would probably not be 'greening the world' but rather
> importing patched
> non-ported modules when we need to dispatch to them. This may sound like
> a big deal, and it is, but it is critical to actually running ported
> code in a real asyncio evenloop.

It will probably require a lot of work to get rid of eventlet. The greenio 
approach is more realistic because projects can be patched one by one, one file 
by one file. The goal is also to run projects unmodified with the greenio 
executor.

> Another interesting problem is (as I have briefly mentioned in [1]) -
> what happens when we need to synchronize between eventlet-run and
> asyncio-run callbacks while we are in the process of porting.

Such issue is solved by greenio. As I wrote, it's not a good idea to have two 
event loops in the same process.

Victor