Open Stack

Hi,

Interesting thread. I have been working on a side project that is a gevent/eventlet replacement [1] that focuses on thread-safety and performance. This came about because of an outstanding bug we have with eventlet not being Thread safe. (We cannot safely enable thread pooling for DB calls so that they will not block.) Unfortunately, I tried to fix the issue while maintaining similar performance but haven’t been completely successful. This led me to believe that it was reasonable to work on an alternative micro-thread implementation on top of greenlet.

So, I admit that this might be somewhat of a biased opinion [2], but I think that using a micro-thread implementation is useful. If not for any other reason, the resulting code is very clean and easy to read. It allows you to write code ‘the normal way’. If you have any sort of experience with real threading, it’s really easy to understand.

Regardless of direction, I would like to see an oslo abstraction so that we can easily switch out the underlying implementation, potentially even making the choice a config option. I think that means that even if we move to asyncio, our abstraction layer provides something that looks like microthreads. I think that it’s maybe the only common ground that makes sense, and it addresses my concerns above regarding readability and ease of use.

- Chris

[1] I haven’t made the code public yet, but will shortly. Mostly I was concerned that it looked like a pile of garbage. :) But it’s at a point that this isn’t a concern anymore.
[2] I really don’t care if my side project is used w/ OpenStack or not, despite thinking we’d do so. It will have usefulness to others outside of OpenStack, even if only for the 80-90% gains in performance that it seems to have compared to eventlet. Most importantly, it has just been fun.


On Feb 4, 2014, at 12:38 PM, victor stinner <victor.stinner at enovance.com> wrote:

> Kevin Conway wrote:
>> Switching our async IO management from eventlet to asyncio would not be a
>> trivial task. Tell me when I'm wrong, but it would require that we
>> completely change our programming model from typical, function-call based
>> programming to use generator-iterators for everything.
> 
> My proposition is to put asyncio on top of greenlet using the greenio project. So the current code can be leaved unchanged (it will continue to eventlet) if you don't want to modify it. New code may use asyncio API instead of greenlet/eventlet API, but the code will still be executed by greenlet. Or you may have different implementations of the same feature, one for eventlet and another for asyncio.
> 
> For example, the Oslo Messaging project has an abstraction of the asynchronous framework called "executor". So you can use a blocking executor, eventlet, trollius or something else. Today, a patch was proposed by Joshua Harlow (*) to support concurrent.futures to use a pool of thread. I don't know yet how asyncio can be integrated in other projects. I'm just starting with Oslo Messaging :-)
> 
> The abstraction layer may be moved from Oslo Messaging to Oslo Incubator, so other projects can reuse it.
> 
> (*) "Start adding a futures executor based executor", https://review.openstack.org/#/c/70914/
> 
> Victor
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev