Open Stack

On 07/09/2014 11:39 AM, Clint Byrum wrote:
> Excerpts from Yuriy Taraday's message of 2014-07-09 03:36:00 -0700:
>> On Tue, Jul 8, 2014 at 11:31 PM, Joshua Harlow <harlowja at yahoo-inc.com>
>> wrote:
>>
>>> I think clints response was likely better than what I can write here, but
>>> I'll add-on a few things,
>>>
>>>
>>>> How do you write such code using taskflow?
>>>>
>>>>   @asyncio.coroutine
>>>>   def foo(self):
>>>>       result = yield from some_async_op(...)
>>>>       return do_stuff(result)
>>>
>>> The idea (at a very high level) is that users don't write this;
>>>
>>> What users do write is a workflow, maybe the following (pseudocode):
>>>
>>> # Define the pieces of your workflow.
>>>
>>> TaskA():
>>>    def execute():
>>>        # Do whatever some_async_op did here.
>>>
>>>    def revert():
>>>        # If execute had any side-effects undo them here.
>>>
>>> TaskFoo():
>>>     ...
>>>
>>> # Compose them together
>>>
>>> flow = linear_flow.Flow("my-stuff").add(TaskA("my-task-a"),
>>> TaskFoo("my-foo"))
>>>
>>
>> I wouldn't consider this composition very user-friendly.
>
> I find it extremely user friendly when I consider that it gives you
> clear lines of delineation between "the way it should work" and "what
> to do when it breaks."

Agreed.

snip...

> Sorry but the code above is nothing like the code that Josh shared. When
> create_network(project) fails, how do we revert its side effects? If we
> want to resume this flow after reboot, how does that work?

Exactly.

> I understand that there is a desire to write everything in beautiful
> python yields, try's, finally's, and excepts. But the reality is that
> python's stack is lost the moment the process segfaults, power goes out
> on that PDU, or the admin rolls out a new kernel.

Yup.

> If we embed taskflow deep in the code, we get those things, and we can
> treat tasks as coroutines and let taskflow's event loop be asyncio just
> the same. If we embed asyncio deep into the code, we don't get any of
> the high level functions and we get just as much code churn.

++

>> There's no limit to coroutine usage. The only problem is the library that
>> would bind everything together.
>> In my example run_task will have to be really smart, keeping track of all
>> started tasks, results of all finished ones, skipping all tasks that have
>> already been done (and substituting already generated results).
>> But all of this is doable. And I find this way of declaring workflows way
>> more understandable than whatever would it look like with Flow.add's
>
> The way the flow is declared is important, as it leads to more isolated
> code. The single place where the flow is declared in Josh's example means
> that the flow can be imported, the state deserialized and inspected,
> and resumed by any piece of code: an API call, a daemon start up, an
> admin command, etc.

Right, this is the main point. We are focusing so much on eventlet vs. 
asyncio, and in doing so we are missing the big picture in how we think 
about the flows of related tasks in our code. Taskflow makes that big 
picture thinking possible, and is what I believe our focus should be on. 
If someone hates seeing eventlet's magic masking of async I/O and wants 
to see Py3K-clean yields, then I think that work belongs in the Taskflow 
engines modules, and not inside Nova directly.

Besides Py3K support and predictive yield points, I haven't seen any 
other valid arguments for spending a bunch of time moving from eventlet 
to asyncio, and certainly no arguments that address the real 
architectural problems inside Nova: that we continue to think at too low 
a level and instead of writing code that naturally groups related sets 
of tasks into workflows, we think instead about how to properly yield 
from one coroutine to another. The point of eventlet, I thought, was to 
hide the low-level stuff so that developers could focus on higher-level 
(and more productive) abstractions. Introducing asyncio contructs into 
the higher level code like Nova and Neutron seems to be a step in the 
wrong direction, IMHO. I'd rather see a renewed focus on getting 
Taskflow incorporated into Nova.

Best,

-jay