Open Stack

+1

To give an example as to why eventlet implicit monkey patch the world isn't especially great (although it's what we are currently using throughout openstack).

The way I think about how it works is to think about what libraries that a single piece of code calls and how it is very hard to predict whether that code will trigger a implicit switch (conceptually similar to a context switch).

Let's take a simple naive piece of code.

>>> import logging
>>> LOG = logging.getLogger(__name__)
>>> LOG.info("hi")

This seems rather straightforward, write 'hi' to some log location. With eventlets implicitness (via ye-old monkey patch everything) it is entirely possible that somewhere inside the logging code there will be a write to a socket (say perhaps this person enabled syslog/socket logger or something like that) and that will block. This causes an implicit switch to another greenthread (and so-on for the applications life-cycle). Now just magnify the amount of understanding required to reason about how the logging library (which is pretty well understood) works with eventlet by the number of libraries in https://github.com/openstack/requirements/blob/master/global-requirements.txt. To understand how all these libraries interact with I/O, threading or other locations where things can implicitly switch is pretty much near impossible. It becomes even more 'hairy' when those libraries themselves acquire some type of locks (did you as an eventlet user remember to monkey patch the threading module?)...

IMHO eventlet has 'seduced' many developers into thinking that it magically makes an application C10K ready even though it easily makes it possible to 'crash and burn' without to much trouble. Is the benefit worth it? Maybe, maybe not...

I'm not saying we should abandon eventlet (likely we can't easily pull this off even if we wanted to), but I do agree that the randomness it provides is not easy to follow, debug, analyze... It gets even more complicated when you start to mix threads (which do exist in python, but are GIL handicapped, although this has been getting better in 3.2+ with GIL improvements) with greenthreads (try figuring out which one is causing race conditions in a gdb session for example).

Anyways, the future of this whole situation looks bright, it will be an interesting balance between making it easy to read/understand (eventlet tries to make it look so easy and no-changes needed, see above seduction) vs. requiring a "big" mind-set change in how libraries and applications are currently written.

Which is the right path to get to the final destination, only time will tell :-)

-Josh

Sent from my really tiny device...

On Feb 6, 2014, at 6:55 PM, "Zane Bitter" <zbitter at redhat.com<mailto:zbitter at redhat.com>> wrote:

On 04/02/14 13:53, Kevin Conway wrote:
On 2/4/14 12:07 PM, "victor stinner"<victor.stinner at enovance.com<mailto:victor.stinner at enovance.com>>  wrote:
>The purpose of replacing eventlet with asyncio is to get a well defined
>control flow, no more surprising task switching at random points.

I disagree with this. Eventlet and gevent yield the execution context
anytime an IO call is made or the 'sleep()' function is called explicitly.
The order in which greenthreads grain execution context is deterministic
even if not entirely obvious. There is no context switching at random.

This is technically correct of course, but in reality there's no way to know whether a particular piece of code is safe from context switches unless you have the entire codebase of the program and all of its libraries in your head at the same time. So no, it's not *random*, but it might as well be. And it's certainly not explicit in the way that asyncio is explicit; it's very much implicit in other operations.

What's more is it shouldn't matter when the context switch happens. When
writing green threaded code you just pretend you have real threads and
understand that things happen in an order other than A => B => C.

If you like pretending you have real threads, you could just use Python threads. Greenthreads exist because people don't want to deal with actual pretend threads.

One of the great benefits of using a green thread abstraction, like
eventlet or gevent, is that it lets you write normal Python code and slap
your concurrency management over the top.

Right, it lets you do that and neglects to mention that it doesn't actually work.

The whole premise of eventlet is that it allows you to write your code without thinking about thread safety, except that you *do* still have to think about thread safety. So the whole reason for its existence is to write cheques that it can't cash. It's conceptually unsound at the most fundamental level.

I'm not suggesting for a second that it would be an easy change - I'm not even sure it would be a good change - but let's not kid ourselves that everything is fine here in happy-land and there's nothing to discuss.

cheers,
Zane.

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140207/c30e264c/attachment.html>