Open Stack

On Nov 24, 2014, at 12:57 PM, Mike Bayer <mbayer at redhat.com> wrote:

> 
>> On Nov 24, 2014, at 12:40 PM, Doug Hellmann <doug at doughellmann.com> wrote:
>> 
>> 
>> This is a good point. I’m not sure we can say “we’ll only use explicit/implicit async in certain cases" because most of our apps actually mix the cases. We have WSGI apps that send RPC messages and we have other apps that receive RPC messages and operate on the database. Can we mix explicit and implicit operating models, or are we going to have to pick one way? If we have to pick one, the implicit model we’re currently using seems more compatible with all of the various libraries and services we depend on, but maybe I’m wrong?
> 
> IMHO, in the ideal case, a single method shouldn’t be mixing calls to a set of database objects as well as calls to RPC APIs at the same time, there should be some kind of method boundary to cross.   There’s a lot of ways to achieve that.

The database calls are inside the method invoked through RPC. System 1 sends an RPC message (call or cast) to system 2 which receives that message and then does something with the database. Frequently “system 1” is an API layer service (mixing WSGI and RPC) and "system 2” is something like the conductor (mixing RPC and DB access).

> 
> What is really needed is some way that code can switch between explicit yields and implicit IO on a per-function basis.   Like a decorator for one or the other.
> 
> The approach that Twisted takes of just using thread pools for those IO-bound elements that aren’t compatible with explicit yields is one way to do this.     This might be the best way to go, if there are in fact issues with mixing in implicit async systems like eventlet.  I can imagine, vaguely, that the eventlet approach of monkey patching might get in the way of things in this more complicated setup.
> 
> Part of what makes this confusing for me is that there’s a lack of clarity over what benefits we’re trying to get from the async work.  If the idea is, the GIL is evil so we need to ban the use of all threads, and therefore must use defer for all IO, then that includes database IO which means we theoretically benefit from eventlet monkeypatching  - in the absence of truly async DBAPIs, this is the only way to have deferrable database IO.
> 
> If the idea instead is, the code we write that deals with messaging would be easier to produce, organize, and understand given an asyncio style approach, but otherwise we aren’t terribly concerned what highly sequential code like database code has to do, then a thread pool may be fine.


A lot of the motivation behind the explicit async changes started as a way to drop our dependency on eventlet because we saw it as blocking our move to Python 3. It is also true that a lot of people don’t like that eventlet monkeypatches system libraries, frequently inconsistently or incorrectly.

Apparently the state of python 3 support for eventlet is a little better than it was when we started talking about this a few years ago, but the monkeypatching is somewhat broken. lifeless suggested trying to fix the monkeypatching, which makes sense. At the summit I think we agreed to continue down the path of supporting both approaches. The issues you’ve raised with using ORMs (or indeed, any IO-based libraries that don’t support explicit async) make me think we should reconsider that discussion with the additional information that didn’t come up in the summit conversation.

Doug

> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev