Open Stack

On 11/23/2014 06:13 PM, Robert Collins wrote:
> On 24 November 2014 at 11:01, victor stinner
> <victor.stinner at enovance.com> wrote:
>> Hi,
>>
>> I'm happy to announce you that I just finished the last piece of the puzzle to add support for trollius coroutines in Oslo Messaging! See my two changes:
>>
>> * Add a new aiogreen executor:
>>   https://review.openstack.org/#/c/136653/
>> * Add an optional executor callback to dispatcher:
>>   https://review.openstack.org/#/c/136652/
>>
>> Related projects:
>>
>> * asyncio is an event loop which is now part of Python 3.4:
>>   http://docs.python.org/dev/library/asyncio.html
>> * trollius is the port of the new asyncio module to Python 2:
>>   http://trollius.readthedocs.org/
>> * aiogreen implements the asyncio API on top of eventlet:
>>   http://aiogreen.readthedocs.org/
>>
>> For the long story and the full history of my work on asyncio in OpenStack since one year, read:
>> http://aiogreen.readthedocs.org/openstack.html
>>
>> The last piece of the puzzle is the new aiogreen project that I released a few days ago. aiogreen is well integrated and fully compatible with eventlet, it can be used in OpenStack without having to modify code. It is almost fully based on trollius, it just has a small glue to reuse eventlet event loop (get read/write notifications of file descriptors).
>>
>> In the past, I tried to use the greenio project, which also implements the asyncio API, but it didn't fit well with eventlet. That's why I wrote a new project.
>>
>> Supporting trollius coroutines in Oslo Messaging is just the first part of the global project. Here is my full plan to replace eventlet with asyncio.
> 
> ...
> 
> So - the technical bits of the plan sound fine.
> 
> On WSGI - if we're in an asyncio world, I don't think WSGI has any
> relevance today - it has no async programming model. While is has
> incremental apis and supports generators, thats not close enough to
> the same thing: so we're going to have to port our glue code to
> whatever container we end up with. As you know I'm pushing on a revamp
> of WSGI right now, and I'd be delighted to help put together a
> WSGI-for-asyncio PEP, but I think its best thought of as a separate
> thing to WSGI per se. It might be a profile of WSGI2 though, since
> there is quite some interest in truely async models.
> 
> However I've a bigger picture concern. OpenStack only relatively
> recently switched away from an explicit async model (Twisted) to
> eventlet.
> 
> I'm worried that this is switching back to something we switched away
> from (in that Twisted and asyncio have much more in common than either
> Twisted and eventlet w/magic, or asyncio and eventlet w/magic).
> 
> If Twisted was unacceptable to the community, what makes asyncio
> acceptable? [Note, I don't really understand why Twisted was moved
> away from, since our problem domain is such a great fit for reactor
> style programming - lots of networking, lots of calling of processes
> that may take some time to complete their work, and occasional DB
> calls [which are equally problematic in eventlet and in
> asyncio/Twisted]. So I'm not arguing against the move, I'm just
> concerned that doing it without addressing whatever the underlying
> thing was, will fail - and I'm also concerned that it will surprise
> folk - since there doesn't seem to be a cross project blueprint
> talking about this fairly fundamental shift in programming model.

I'm not going to comment on the pros and cons - I think we all know I'm
a fan of threads. But I have been around a while, so - for those who
haven't been:

When we started the project, nova used twisted and swift used eventlet.
As we've consistently endeavored to not have multiple frameworks, we
entered in to the project's first big flame war:

"twisted vs. eventlet"

It was _real_ fun, I promise. But a the heart was a question of whether
we were going to rewrite swift in twisted or rewrite nova in eventlet.

The main 'winning' answer came down to twisted being very opaque for new
devs - while it's very powerful for experienced devs, we decided to opt
for eventlet which does not scare new devs with a completely different
programming model. (reactors and deferreds and whatnot)

Now, I wouldn't say we _just_ ported from Twisted, I think we finished
that work about 4 years ago. :)

Monty