[openstack-dev] [oslo] Asyncio and oslo.messaging

Clint Byrum clint at fewbar.com
Tue Jul 8 18:43:01 UTC 2014


Excerpts from Victor Stinner's message of 2014-07-08 05:47:36 -0700:
> Hi Joshua,
> 
> You asked a lot of questions. I will try to answer.
> 
> Le lundi 7 juillet 2014, 17:41:34 Joshua Harlow a écrit :
> > * Why focus on a replacement low level execution model integration instead
> > of higher level workflow library or service (taskflow, mistral... other)
> > integration?
> 
> I don't know tasklow, I cannot answer to this question.
> 
> How do you write such code using taskflow?
> 
>   @asyncio.coroutine
>   def foo(self):
>       result = yield from some_async_op(...)
>       return do_stuff(result)
> 

Victor, this is a low level piece of code, which highlights the problem
that taskflow's higher level structure is meant to address. In writing
OpenStack, we want to accomplish tasks based on a number of events. Users,
errors, etc. We don't explicitly want to run coroutines, we want to
attach volumes, spawn vms, and store files.

See this:

http://docs.openstack.org/developer/taskflow/examples.html

The result is consumed in the next task in the flow. Meanwhile we get
a clear definition of work-flow and very clear methods for resumption,
retry, etc. So the expression is not as tightly bound as the code above,
but that is the point, because we want to break things up into tasks
which are clearly defined and then be able to resume each one
individually.

So what I think Josh is getting at, is that we could add asyncio support
into taskflow as an abstraction for tasks that want to be non-blocking,
and then we can focus on refactoring the code around high level work-flow
expression rather than low level asyncio and coroutines.

> > * Was the heat (asyncio-like) execution model[1] examined and learned from
> > before considering moving to asyncio?
> 
> I looked at Heat coroutines, but it has a design very different from asyncio.
> 
> In short, asyncio uses an event loop running somewhere in the background, 
> whereas Heat explicitly schedules the execution of some tasks (with 
> "TaskRunner"), blocks until it gets the result and then stop completly its 
> "event loop". It's possible to implement that with asyncio, there is for 
> example a run_until_complete() method stopping the event loop when a future is 
> done. But asyncio event loop is designed to run "forever", so various projects 
> can run tasks "at the same time", not only a very specific section of the code 
> to run a set of tasks.
> 
> asyncio is not only designed to schedule callbacks, it's also designed to 
> manager file descriptors (especially sockets). It can also spawn and manager 
> subprocessses. This is not supported by Heat scheduler.
> 
> IMO Heat scheduler is too specific, it cannot be used widely in OpenStack.
> 

This is sort of backwards to what Josh was suggesting. Heat can't continue
with the current approach, which is coroutine based, because we need the
the execution stack to not be in RAM on a single engine. We are going
to achieve even more concurrency than we have now through an even higher
level of task abstraction as part of the move to a convergence model. We
will likely use task-flow to express these tasks so that they are more
resumable and generally resilient to failure.

> > Along a related question,
> > seeing that openstack needs to support py2.x and py3.x will this mean that
> > trollius will be required to be used in 3.x (as it is the least common
> > denominator, not new syntax like 'yield from' that won't exist in 2.x).
> > Does this mean that libraries that will now be required to change will be
> > required to use trollius (the pulsar[6] framework seemed to mesh these two
> > nicely); is this understood by those authors?
> 
> It *is* possible to write code working on asyncio and trollius:
> http://trollius.readthedocs.org/#write-code-working-on-trollius-and-tulip
> 
> They are different options for that. They are already projects supporting 
> asyncio and trollius.
> 
> > Is this the direction we
> > want to go down (if we stay focused on ensuring py3.x compatible, then why
> > not just jump to py3.x in the first place)?
> 
> FYI OpenStack does not support Python 3 right now. I'm working on porting 
> OpenStack to Python 3, we made huge progress, but it's not done yet.
> 
> Anyway, the new RHEL 7 release doesn't provide Python 3.3 in the default 
> system, you have to enable the SCL repository (which provides Python 3.3). And 
> Python 2.7 or even 2.6 is still used in production.
> 
> I would also prefer to use directly "yield from" and "just" drop Python 2 
> support. But dropping Python 2 support is not going to happen before at least 
> 2 years.
> 

Long term porting is important, however, we have immediate needs for
improvements in resilience and scalability. We cannot hang _any_ of that
on Python 3.



More information about the OpenStack-dev mailing list