Open Stack

Hi Joshua,

You asked a lot of questions. I will try to answer.

Le lundi 7 juillet 2014, 17:41:34 Joshua Harlow a écrit :
> * Why focus on a replacement low level execution model integration instead
> of higher level workflow library or service (taskflow, mistral... other)
> integration?

I don't know tasklow, I cannot answer to this question.

How do you write such code using taskflow?

  @asyncio.coroutine
  def foo(self):
      result = yield from some_async_op(...)
      return do_stuff(result)


> * Was the heat (asyncio-like) execution model[1] examined and learned from
> before considering moving to asyncio?

I looked at Heat coroutines, but it has a design very different from asyncio.

In short, asyncio uses an event loop running somewhere in the background, 
whereas Heat explicitly schedules the execution of some tasks (with 
"TaskRunner"), blocks until it gets the result and then stop completly its 
"event loop". It's possible to implement that with asyncio, there is for 
example a run_until_complete() method stopping the event loop when a future is 
done. But asyncio event loop is designed to run "forever", so various projects 
can run tasks "at the same time", not only a very specific section of the code 
to run a set of tasks.

asyncio is not only designed to schedule callbacks, it's also designed to 
manager file descriptors (especially sockets). It can also spawn and manager 
subprocessses. This is not supported by Heat scheduler.

IMO Heat scheduler is too specific, it cannot be used widely in OpenStack.


>   * A side-question, how do asyncio and/or trollius support debugging, do
> they support tracing individual co-routines? What about introspecting the
> state a coroutine has associated with it? Eventlet at least has
> http://eventlet.net/doc/modules/debug.html (which is better than nothing);
> does an equivalent exist?

asyncio documentation has a section dedicated to debug:
https://docs.python.org/dev/library/asyncio-dev.html

asyncio.Task has get_stack() and print_stack() methods can be used to get the 
current state of a task. I don't know if it is what you are looking for.

I modified recently asyncio Task and Handle to save the traceback where they 
were created, it's now much easier to debug code. asyncio now also logs "slow 
callbacks" which may block the event loop (reduce the reactivity).

Read: We are making progress to ease the development and debug asyncio code.

I don't know exactly what you expect for debugging. However, there is no 
"tracing" feature" in asyncio yet.


> This is the part that I really wonder about. Since asyncio isn't just a
> drop-in replacement for eventlet (which hid the async part under its
> *black magic*), I very much wonder how the community will respond to this
> kind of mindset change (along with its new *black magic*).

I disagree with you, asyncio is not "black magic". It's well defined. The 
execution of a coroutine is complex, but it doesn't use magic. IMO eventlet 
task switching is more black magic, it's not possible to guess it just by 
reading the code.

Sorry but asyncio is nothing new :-( It's just a fresh design based on 
previous projects.

Python has Twisted since more than 10 years. More recently, Tornado came. Both 
support coroutines using "yield" (see @inlineCallbacks and toro). Thanks to 
these two projects, there are already libraries which have an async API, using 
coroutines or callbacks.

These are just the two major projects, they are much more projects, but they 
are smaller and younger.

Other programming languages are also moving to asynchronous programming. Read 
for example this article which lists some of them:
https://glyph.twistedmatrix.com/2014/02/unyielding.html


> * Is the larger python community ready for this?
> 
> Seeing other responses for supporting libraries that aren't asyncio
> compatible it doesn't inspire confidence that this path is ready to be
> headed down. Partially this is due to the fact that its a completely new
> programming model and alot of underlying libraries will be forced to
> change to accommodate this (sqlalchemy, others listed in [5]...).

The design of the asyncio core is really simple, it's mostly based on 
callbacks. Callbacks is an old concept, and many libraries already accept a 
callback to send the result when they are done.

In asyncio, you can use loop.call_soon(callback) to schedule the execution of 
a callback, and future.add_done_callback() to connect a future to a callback.

Oh by way, the core of Twisted and Tornado also uses callbacks. Using 
callbacks makes you compatible with Twisted, Tornado and asyncio.


asyncio already has projects compatible with it, see this list:
https://code.google.com/p/tulip/wiki/ThirdParty

There are for example Redis, PostgreSQL and MongoDB drivers. There is also an 
AMQP implementation.


You are not forced to make your code async. You can run blocking code in 
threads (a pool of threads) using loop.run_in_executor(). For example, DNS 
resolution is currently done like that (the socket.getaddrinfo() function is 
blocking). Or you also just block until you get the result, in some cases it 
does not impact performances (ex: CLI client waiting for the answer of a 
server).


> Along a related question,
> seeing that openstack needs to support py2.x and py3.x will this mean that
> trollius will be required to be used in 3.x (as it is the least common
> denominator, not new syntax like 'yield from' that won't exist in 2.x).
> Does this mean that libraries that will now be required to change will be
> required to use trollius (the pulsar[6] framework seemed to mesh these two
> nicely); is this understood by those authors?

It *is* possible to write code working on asyncio and trollius:
http://trollius.readthedocs.org/#write-code-working-on-trollius-and-tulip

They are different options for that. They are already projects supporting 
asyncio and trollius.


> Is this the direction we
> want to go down (if we stay focused on ensuring py3.x compatible, then why
> not just jump to py3.x in the first place)?

FYI OpenStack does not support Python 3 right now. I'm working on porting 
OpenStack to Python 3, we made huge progress, but it's not done yet.

Anyway, the new RHEL 7 release doesn't provide Python 3.3 in the default 
system, you have to enable the SCL repository (which provides Python 3.3). And 
Python 2.7 or even 2.6 is still used in production.

I would also prefer to use directly "yield from" and "just" drop Python 2 
support. But dropping Python 2 support is not going to happen before at least 
2 years.

We should port OpenStack to Python 3 right now!
http://techs.enovance.com/6521/openstack_python3

Status of the port to Python 3:
https://wiki.openstack.org/wiki/Python3


I hope that I replied to some of your questions.

Victor