Open Stack

Thanks clint, that was the gist of what I was getting at with the (likely
to long) email.

-Josh

-----Original Message-----
From: Clint Byrum <clint at fewbar.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Tuesday, July 8, 2014 at 11:43 AM
To: openstack-dev <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

>Excerpts from Victor Stinner's message of 2014-07-08 05:47:36 -0700:
>> Hi Joshua,
>> 
>> You asked a lot of questions. I will try to answer.
>> 
>> Le lundi 7 juillet 2014, 17:41:34 Joshua Harlow a écrit :
>> > * Why focus on a replacement low level execution model integration
>>instead
>> > of higher level workflow library or service (taskflow, mistral...
>>other)
>> > integration?
>> 
>> I don't know tasklow, I cannot answer to this question.
>> 
>> How do you write such code using taskflow?
>> 
>>   @asyncio.coroutine
>>   def foo(self):
>>       result = yield from some_async_op(...)
>>       return do_stuff(result)
>> 
>
>Victor, this is a low level piece of code, which highlights the problem
>that taskflow's higher level structure is meant to address. In writing
>OpenStack, we want to accomplish tasks based on a number of events. Users,
>errors, etc. We don't explicitly want to run coroutines, we want to
>attach volumes, spawn vms, and store files.
>
>See this:
>
>http://docs.openstack.org/developer/taskflow/examples.html
>
>The result is consumed in the next task in the flow. Meanwhile we get
>a clear definition of work-flow and very clear methods for resumption,
>retry, etc. So the expression is not as tightly bound as the code above,
>but that is the point, because we want to break things up into tasks
>which are clearly defined and then be able to resume each one
>individually.
>
>So what I think Josh is getting at, is that we could add asyncio support
>into taskflow as an abstraction for tasks that want to be non-blocking,
>and then we can focus on refactoring the code around high level work-flow
>expression rather than low level asyncio and coroutines.
>
>> > * Was the heat (asyncio-like) execution model[1] examined and learned
>>from
>> > before considering moving to asyncio?
>> 
>> I looked at Heat coroutines, but it has a design very different from
>>asyncio.
>> 
>> In short, asyncio uses an event loop running somewhere in the
>>background, 
>> whereas Heat explicitly schedules the execution of some tasks (with
>> "TaskRunner"), blocks until it gets the result and then stop completly
>>its 
>> "event loop". It's possible to implement that with asyncio, there is
>>for 
>> example a run_until_complete() method stopping the event loop when a
>>future is 
>> done. But asyncio event loop is designed to run "forever", so various
>>projects 
>> can run tasks "at the same time", not only a very specific section of
>>the code 
>> to run a set of tasks.
>> 
>> asyncio is not only designed to schedule callbacks, it's also designed
>>to 
>> manager file descriptors (especially sockets). It can also spawn and
>>manager 
>> subprocessses. This is not supported by Heat scheduler.
>> 
>> IMO Heat scheduler is too specific, it cannot be used widely in
>>OpenStack.
>> 
>
>This is sort of backwards to what Josh was suggesting. Heat can't continue
>with the current approach, which is coroutine based, because we need the
>the execution stack to not be in RAM on a single engine. We are going
>to achieve even more concurrency than we have now through an even higher
>level of task abstraction as part of the move to a convergence model. We
>will likely use task-flow to express these tasks so that they are more
>resumable and generally resilient to failure.
>
>> > Along a related question,
>> > seeing that openstack needs to support py2.x and py3.x will this mean
>>that
>> > trollius will be required to be used in 3.x (as it is the least common
>> > denominator, not new syntax like 'yield from' that won't exist in
>>2.x).
>> > Does this mean that libraries that will now be required to change
>>will be
>> > required to use trollius (the pulsar[6] framework seemed to mesh
>>these two
>> > nicely); is this understood by those authors?
>> 
>> It *is* possible to write code working on asyncio and trollius:
>> 
>>http://trollius.readthedocs.org/#write-code-working-on-trollius-and-tulip
>> 
>> They are different options for that. They are already projects
>>supporting 
>> asyncio and trollius.
>> 
>> > Is this the direction we
>> > want to go down (if we stay focused on ensuring py3.x compatible,
>>then why
>> > not just jump to py3.x in the first place)?
>> 
>> FYI OpenStack does not support Python 3 right now. I'm working on
>>porting 
>> OpenStack to Python 3, we made huge progress, but it's not done yet.
>> 
>> Anyway, the new RHEL 7 release doesn't provide Python 3.3 in the
>>default 
>> system, you have to enable the SCL repository (which provides Python
>>3.3). And 
>> Python 2.7 or even 2.6 is still used in production.
>> 
>> I would also prefer to use directly "yield from" and "just" drop Python
>>2 
>> support. But dropping Python 2 support is not going to happen before at
>>least 
>> 2 years.
>> 
>
>Long term porting is important, however, we have immediate needs for
>improvements in resilience and scalability. We cannot hang _any_ of that
>on Python 3.
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev