[openstack-dev] Asynchrounous programming: replace eventlet with asyncio

Chuck Thier cthier at gmail.com
Fri Feb 7 16:38:48 UTC 2014


Concurrency is hard, let's blame the tools!

Any lib that we use in python is going to have a set of trade-offs.
 Looking at a couple of the options on the table:

1.  Threads:  Great! code doesn't have to change too much, but now that
code *will* be preempted at any time, so now we have to worry about locking
and we have even more race conditions that are difficult to debug.

2.  Asyncio:  Explicit FTW!  Except now that big list of dependencies has
to also support the same form of explicit concurrency.  This is a trade-off
that twisted makes as well.  Any library that might block has to have a
separate library made for it.

We could dig deeper, but hopefully you see what I mean.  Changing tools may
solve one problem, but at the same time introduce a different set of
problems.

I think the biggest issue with using Eventlet is that developers want to
treat it like magic, and you can't do that.  If you are monkey patching the
world, then you are doing it wrong.  How about we take a moment to learn
how to use the tools we have effectively, rather than just blaming them.
 Many projects have managed to use Eventlet effectively (including some in
Openstack).

Eventlet isn't perfect, but it has gotten us quite a ways.  If you do
choose to use another library, please make sure you are trading for the
right set of problems.

--
Chuck


On Fri, Feb 7, 2014 at 1:07 AM, Joshua Harlow <harlowja at yahoo-inc.com>wrote:

>  +1
>
> To give an example as to why eventlet implicit monkey patch the world
> isn't especially great (although it's what we are currently using
> throughout openstack).
>
> The way I think about how it works is to think about what libraries that a
> single piece of code calls and how it is very hard to predict whether that
> code will trigger a implicit switch (conceptually similar to a context
> switch).
>
> Let's take a simple naive piece of code.
>
> >>> import logging
> >>> LOG = logging.getLogger(__name__)
> >>> LOG.info("hi")
>
> This seems rather straightforward, write 'hi' to some log location. With
> eventlets implicitness (via ye-old monkey patch everything) it is entirely
> possible that somewhere inside the logging code there will be a write to a
> socket (say perhaps this person enabled syslog/socket logger or something
> like that) and that will block. This causes an implicit switch to another
> greenthread (and so-on for the applications life-cycle). Now just magnify
> the amount of understanding required to reason about how the logging
> library (which is pretty well understood) works with eventlet by the number
> of libraries in
> https://github.com/openstack/requirements/blob/master/global-requirements.txt.
> To understand how all these libraries interact with I/O, threading or other
> locations where things can implicitly switch is pretty much near
> impossible. It becomes even more 'hairy' when those libraries themselves
> acquire some type of locks (did you as an eventlet user remember to monkey
> patch the threading module?)...
>
> IMHO eventlet has 'seduced' many developers into thinking that it
> magically makes an application C10K ready even though it easily makes it
> possible to 'crash and burn' without to much trouble. Is the benefit worth
> it? Maybe, maybe not...
>
> I'm not saying we should abandon eventlet (likely we can't easily pull
> this off even if we wanted to), but I do agree that the randomness it
> provides is not easy to follow, debug, analyze... It gets even more
> complicated when you start to mix threads (which do exist in python, but
> are GIL handicapped, although this has been getting better in 3.2+ with GIL
> improvements) with greenthreads (try figuring out which one is causing race
> conditions in a gdb session for example).
>
> Anyways, the future of this whole situation looks bright, it will be an
> interesting balance between making it easy to read/understand (eventlet
> tries to make it look so easy and no-changes needed, see above seduction)
> vs. requiring a "big" mind-set change in how libraries and applications are
> currently written.
>
> Which is the right path to get to the final destination, only time will
> tell :-)
>
>  -Josh
>
>
> Sent from my really tiny device...
>
> On Feb 6, 2014, at 6:55 PM, "Zane Bitter" <zbitter at redhat.com> wrote:
>
>  On 04/02/14 13:53, Kevin Conway wrote:
>
> On 2/4/14 12:07 PM, "victor stinner"<victor.stinner at enovance.com>  wrote:
>
>  >The purpose of replacing eventlet with asyncio is to get a well defined
>
>  >control flow, no more surprising task switching at random points.
>
>
>  I disagree with this. Eventlet and gevent yield the execution context
>
> anytime an IO call is made or the 'sleep()' function is called explicitly.
>
> The order in which greenthreads grain execution context is deterministic
>
> even if not entirely obvious. There is no context switching at random.
>
>
> This is technically correct of course, but in reality there's no way to
> know whether a particular piece of code is safe from context switches
> unless you have the entire codebase of the program and all of its libraries
> in your head at the same time. So no, it's not *random*, but it might as
> well be. And it's certainly not explicit in the way that asyncio is
> explicit; it's very much implicit in other operations.
>
> What's more is it shouldn't matter when the context switch happens. When
>
> writing green threaded code you just pretend you have real threads and
>
> understand that things happen in an order other than A => B => C.
>
>
> If you like pretending you have real threads, you could just use Python
> threads. Greenthreads exist because people don't want to deal with actual
> pretend threads.
>
> One of the great benefits of using a green thread abstraction, like
>
> eventlet or gevent, is that it lets you write normal Python code and slap
>
> your concurrency management over the top.
>
>
> Right, it lets you do that and neglects to mention that it doesn't
> actually work.
>
> The whole premise of eventlet is that it allows you to write your code
> without thinking about thread safety, except that you *do* still have to
> think about thread safety. So the whole reason for its existence is to
> write cheques that it can't cash. It's conceptually unsound at the most
> fundamental level.
>
> I'm not suggesting for a second that it would be an easy change - I'm not
> even sure it would be a good change - but let's not kid ourselves that
> everything is fine here in happy-land and there's nothing to discuss.
>
> cheers,
> Zane.
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140207/bbd08f29/attachment.html>


More information about the OpenStack-dev mailing list