Open Stack

> On Nov 23, 2014, at 6:30 PM, Monty Taylor <mordred at inaugust.com> wrote:
> 
> On 11/23/2014 06:13 PM, Robert Collins wrote:
>> On 24 November 2014 at 11:01, victor stinner
>> <victor.stinner at enovance.com> wrote:
>>> Hi,
>>> 
>>> I'm happy to announce you that I just finished the last piece of the puzzle to add support for trollius coroutines in Oslo Messaging! See my two changes:
>>> 
>>> * Add a new aiogreen executor:
>>>  https://review.openstack.org/#/c/136653/
>>> * Add an optional executor callback to dispatcher:
>>>  https://review.openstack.org/#/c/136652/
>>> 
>>> Related projects:
>>> 
>>> * asyncio is an event loop which is now part of Python 3.4:
>>>  http://docs.python.org/dev/library/asyncio.html
>>> * trollius is the port of the new asyncio module to Python 2:
>>>  http://trollius.readthedocs.org/
>>> * aiogreen implements the asyncio API on top of eventlet:
>>>  http://aiogreen.readthedocs.org/
>>> 
>>> For the long story and the full history of my work on asyncio in OpenStack since one year, read:
>>> http://aiogreen.readthedocs.org/openstack.html
>>> 
>>> The last piece of the puzzle is the new aiogreen project that I released a few days ago. aiogreen is well integrated and fully compatible with eventlet, it can be used in OpenStack without having to modify code. It is almost fully based on trollius, it just has a small glue to reuse eventlet event loop (get read/write notifications of file descriptors).
>>> 
>>> In the past, I tried to use the greenio project, which also implements the asyncio API, but it didn't fit well with eventlet. That's why I wrote a new project.
>>> 
>>> Supporting trollius coroutines in Oslo Messaging is just the first part of the global project. Here is my full plan to replace eventlet with asyncio.
>> 
>> ...
>> 
>> So - the technical bits of the plan sound fine.
>> 
>> On WSGI - if we're in an asyncio world, I don't think WSGI has any
>> relevance today - it has no async programming model. While is has
>> incremental apis and supports generators, thats not close enough to
>> the same thing: so we're going to have to port our glue code to
>> whatever container we end up with. As you know I'm pushing on a revamp
>> of WSGI right now, and I'd be delighted to help put together a
>> WSGI-for-asyncio PEP, but I think its best thought of as a separate
>> thing to WSGI per se. It might be a profile of WSGI2 though, since
>> there is quite some interest in truely async models.
>> 
>> However I've a bigger picture concern. OpenStack only relatively
>> recently switched away from an explicit async model (Twisted) to
>> eventlet.
>> 
>> I'm worried that this is switching back to something we switched away
>> from (in that Twisted and asyncio have much more in common than either
>> Twisted and eventlet w/magic, or asyncio and eventlet w/magic).
>> 
>> If Twisted was unacceptable to the community, what makes asyncio
>> acceptable? [Note, I don't really understand why Twisted was moved
>> away from, since our problem domain is such a great fit for reactor
>> style programming - lots of networking, lots of calling of processes
>> that may take some time to complete their work, and occasional DB
>> calls [which are equally problematic in eventlet and in
>> asyncio/Twisted]. So I'm not arguing against the move, I'm just
>> concerned that doing it without addressing whatever the underlying
>> thing was, will fail - and I'm also concerned that it will surprise
>> folk - since there doesn't seem to be a cross project blueprint
>> talking about this fairly fundamental shift in programming model.
> 
> I'm not going to comment on the pros and cons - I think we all know I'm
> a fan of threads. But I have been around a while, so - for those who
> haven't been:
> 
> When we started the project, nova used twisted and swift used eventlet.
> As we've consistently endeavored to not have multiple frameworks, we
> entered in to the project's first big flame war:
> 
> "twisted vs. eventlet"
> 
> It was _real_ fun, I promise. But a the heart was a question of whether
> we were going to rewrite swift in twisted or rewrite nova in eventlet.
> 
> The main 'winning' answer came down to twisted being very opaque for new
> devs - while it's very powerful for experienced devs, we decided to opt
> for eventlet which does not scare new devs with a completely different
> programming model. (reactors and deferreds and whatnot)
> 
> Now, I wouldn't say we _just_ ported from Twisted, I think we finished
> that work about 4 years ago. :)
> 

For whatever it’s worth, I find explicit async io to be _way_ easier to
understand for the same reason I find threaded code to be a rats nest.

The co-routine style of asyncio (or Twisted’s inlineCallbacks) solves
almost all of the problems that I think most people have with explicit
asyncio (namely the callback hell) while still getting the benefits.

Glyph wrote a good post that mirrors my opinions on implicit vs explicit
here: https://glyph.twistedmatrix.com/2014/02/unyielding.html.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA