Open Stack

Excerpts from Joshua Harlow's message of 2014-07-07 10:41:34 -0700:
> So I've been thinking how to respond to this email, and here goes (shields
> up!),
> 
> First things first; thanks mark and victor for the detailed plan and
> making it visible to all. It's very nicely put together and the amount of
> thought put into it is great to see. I always welcome an effort to move
> toward a new structured & explicit programming model (which asyncio
> clearly helps make possible and strongly encourages/requires).
> 

I too appreciate the level of detail in the proposal. I think I
understand where it wants to go.

> So now to some questions that I've been thinking about how to
> address/raise/ask (if any of these appear as FUD, they were not meant to
> be):
> 
> * Why focus on a replacement low level execution model integration instead
> of higher level workflow library or service (taskflow, mistral... other)
> integration?
> 
> Since pretty much all of openstack is focused around workflows that get
> triggered by some API activated by some user/entity having a new execution
> model (asyncio) IMHO doesn't seem to be shifting the needle in the
> direction that improves the scalability, robustness and crash-tolerance of
> those workflows (and the associated projects those workflows are currently
> defined & reside in). I *mostly* understand why we want to move to asyncio
> (py3, getting rid of eventlet, better performance? new awesomeness...) but
> it doesn't feel that important to actually accomplish seeing the big holes
> that openstack has right now with scalability, robustness... Let's imagine
> a different view on this; if all openstack projects declaratively define
> the workflows there APIs trigger (nova is working on task APIs, cinder is
> getting there to...), and in the future when the projects are *only*
> responsible for composing those workflows and handling the API inputs &
> responses then the need for asyncio or other technology can move out from
> the individual projects and into something else (possibly something that
> is being built & used as we speak). With this kind of approach the
> execution model can be an internal implementation detail of the workflow
> 'engine/processor' (it will also be responsible for fault-tolerant, robust
> and scalable execution). If this seems reasonable, then why not focus on
> integrating said thing into openstack and move the projects to a model
> that is independent of eventlet, asyncio (or the next greatest thing)
> instead? This seems to push the needle in the right direction and IMHO
> (and hopefully others opinions) has a much bigger potential to improve the
> various projects than just switching to a new underlying execution model.
> 
> * Was the heat (asyncio-like) execution model[1] examined and learned from
> before considering moving to asyncio?
> 
> I will try not to put words into the heat developers mouths (I can't do it
> justice anyway, hopefully they can chime in here) but I believe that heat
> has a system that is very similar to asyncio and coroutines right now and
> they are actively moving to a different model due to problems in part due
> to using that coroutine model in heat. So if they are moving somewhat away
> from that model (to a more declaratively workflow model that can be
> interrupted and converged upon [2]) why would it be beneficial for other
> projects to move toward the model they are moving away from (instead of
> repeating the issues the heat team had with coroutines, ex, visibility
> into stack/coroutine state, scale limitations, interruptibility...)?
> 

I'd like to hear Zane's opinions as he developed the rather light weight
code that we use. It has been quite a learning curve for me but I do
understand how to use the task scheduler we have in Heat now.

Heat's model is similar to asyncio, but is entirely limited in scope. I
think it has stayed relatively manageable because it is really only used
for a few explicit tasks where a high degree of concurrency makes a lot
of sense. We are not using it for I/O concurrency (eventlet still does
that) but rather for request concurrency. So we tell nova to boot 100
servers with 100 coroutines that have 100 other coroutines to block
further execution until those servers are active. We are by no means
using it as a general purpose concurrency programming model.

That said, as somebody working on the specification to move toward a
more taskflow-like (perhaps even entirely taskflow-based) model in Heat,
I think that is the way to go. The fact that we already have an event
loop that doesn't need to be explicit except at the very lowest levels
makes me want to keep that model. And we clearly need help with how to
define workflows, which something like taskflow will do nicely.

>   
>   * A side-question, how do asyncio and/or trollius support debugging, do
> they support tracing individual co-routines? What about introspecting the
> state a coroutine has associated with it? Eventlet at least has
> http://eventlet.net/doc/modules/debug.html (which is better than nothing);
> does an equivalent exist?
> 

I second that question. My twisted experience has left me very
suspicious of coroutines and debugging.

> * What's the current thinking on avoiding the chaos (code-change-wise and
> brain-power-wise) that will come from a change to asyncio?
> 
> This is the part that I really wonder about. Since asyncio isn't just a
> drop-in replacement for eventlet (which hid the async part under its
> *black magic*), I very much wonder how the community will respond to this
> kind of mindset change (along with its new *black magic*). Will the
> TC/foundation offer training, tutorials... on the change that this brings?
> Should the community even care? If we say just focus on workflows & let
> the workflow 'engine/processor' do the dirty work; then I believe the
> community really doesn't need to care (and rightfully so) about how their
> workflows get executed (either by taskflow, mistral, pigeons...). I
> believe this seems like a fair assumption to make; it could even be
> reinforced (I am not an expert here) with the defcore[4] work that seems
> to be standardizing the integration tests that verify those workflows (and
> associated APIs) act as expected in the various commercial implementations.
> 

Angus summed that up nicely in his sub-thread about just sticking with
eventlet. I'm concerned as well.

> * Is the larger python community ready for this?
> 
> Seeing other responses for supporting libraries that aren't asyncio
> compatible it doesn't inspire confidence that this path is ready to be
> headed down. Partially this is due to the fact that its a completely new
> programming model and alot of underlying libraries will be forced to
> change to accommodate this (sqlalchemy, others listed in [5]...). Do
> others feel it's appropriate to start this at this time, or does it feel
> premature? Of course we have to start somewhere but I start to wonder if
> effort is better spent elsewhere (see above). Along a related question,
> seeing that openstack needs to support py2.x and py3.x will this mean that
> trollius will be required to be used in 3.x (as it is the least common
> denominator, not new syntax like 'yield from' that won't exist in 2.x).
> Does this mean that libraries that will now be required to change will be
> required to use trollius (the pulsar[6] framework seemed to mesh these two
> nicely); is this understood by those authors? Is this the direction we
> want to go down (if we stay focused on ensuring py3.x compatible, then why
> not just jump to py3.x in the first place)?
> 

I haven't seen any language fully embrace coroutine based systems, but
I'm not looking very closely at other languages lately.

Python may be a pioneer in this area, I'm not sure. But what I see is
the opposite: light weight coordinated threading systems seem to be
quite popular, and I think not just a little bit because they are easier
to debug.