Open Stack

Johannes Erdfelt <johannes at erdfelt.com> wrote:

> On Wed, Jan 28, 2015, Mike Bayer <mbayer at redhat.com> wrote:
>> I can envision turning this driver into a total monster, adding
>> C-speedups where needed but without getting in the way of async
>> patching, adding new APIs for explicit async, and everything else.
>> However, I’ve no idea what the developers have an appetite for.
> 
> This is great information. I appreciate the work on evaluating it.
> 
> Can I bring up the alternative of dropping eventlet and switching to
> native threads?

I'd be your best friend for life. But I don’t think you’re going to get much
traction here on that. There’s three camps on this issue: the
threads-are-fine camp, the “everything should be async! you should never
trust threads / IO to run implicitly!” camp, and the “we need 1000
concurrent connections but don’t want to get into explicit async so let’s
use eventlet/gevent”.

Myself, I am in a fourth camp, use explicit async for those kinds of logic
where it makes sense from a code organization point of view, and not for
those where it does not. The use case of gevent/eventlet is itself in somewhat
of a fringe camp, the one where you need many hundreds of concurrent database
connections within one process for something (which you do not).

The kinds of use cases that are very valid for explicit async are things
like code that is heavy on web service requests and asynchronous message
queues. However, code that deals with databases and transactions really
gains nothing and loses everything by being muddied and complicated with
this very awkward approach.

Unfortunately, the “debate” as I continue to follow it is really not about these things.  
From my observations, it seems to revolve around the following points:

A. async code is “faster” (it is not)

B. threads are “incorrect” (they are not)

C. code that performs IO without the programmer setting aside
explicit directives, deferrals, and other whatnot to accommodate this event is “wrong”
(it is not).

As long as the debate is over these largely ephemeral assertions, which keep coming
up over and over again, Openstack has stuck its feet in the mud with the eventlet 
thing as a kind of compromise (even though crowd C is not appeased by this).


> We spend a lot of time working on the various incompatibilies between
> eventlet and other libraries we use. It also restricts us by making it
> difficult to use an entire class of python modules (that use C
> extensions for performance, etc).
> 
> I personally have spent more time than I wish to admit fixing bugs in
> eventlet and troubleshooting problems we've had.
> 
> And it's never been clear to me why we *need* to use eventlet or
> green threads in general.
> 
> Our modern Nova appears to only be weakly tied to eventlet and greenlet.
> I think we would spend less time replacing eventlet with native threads
> than we'll spend in the future trying to fit our code and dependencies
> into the eventlet shaped hole we currently have.
> 
> I'm not as familiar with the code in other OpenStack projects, but from
> what I have seen, they appear to be similar to Nova and are only weakly
> tied to eventlet/greenlet.
> 
> JE
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev