Open Stack

> Then, we can make task executor interface public and allow clients to
> provide their own task executors. It will be possible then for Mistral
> to implement its own task executor, or several, and share the
> executors between all the engine instances.

I'm afraid that if we start to tear apart the TaskFlow engine, it would quickly become a mess to support. Besides, the amount of things left to integrate after we throw out engine might be so low it proof the whole process of integration to be just nominal and we are back to square one. Any way, task execution is the part that least bothers me, both graph action and the engine itself is where the pain will be.

> That is part of our public API, it is stable and good enough. Basically,
> I don't think this API needs any major change.


> But whatever should and will be done about it, I daresay all that work
> can be done without affecting API more then I described above.



I completely agree that we should not change the public API of the sync engine, especially the one in helpers. What we need is, on the contrary, a low level construct that would do the number of things i stated previously, but will be a part of public API of TaskFlow so we can be sure it would work exactly the same way it worked yesterday.

--  
Kirill Izotov


пятница, 4 апреля 2014 г. в 2:04, Ivan Melnikov написал:

>  
> I'm trying to catch up this rather long and interesting discussion,
> sorry for somewhat late reply.
>  
> I can see aspects of 'lazy model' support in TaskFlow:
> - how tasks are executed and reverted
> - how flows are run
> - how engine works internally
>  
> Let me address those aspects separately.
>  
> == Executing and reverting tasks ==
>  
> I think that should be done via different interface then running a flow
> (or scheduling it to run), as it is completely different thing. In
> current TaskFlow this interface is called task executor:
> https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57
>  
> That is actually how our WorkerBasedEngine was implemented: it's the
> same engine with special task executor that schedules tasks on worker
> instead of running task code locally.
>  
> Task executors are not aware of flows by design, all they do is
> executing and reverting tasks. That means that task executors can be
> easily shared between engines if that's wanted.
>  
> Current TaskExecutorBase interface uses futures (PEP 3148-like). When I
> proposed it, futures looked like good tool for the task at hand (see
> e.g. async task etherpad
> https://etherpad.openstack.org/p/async-taskflow-tasks)
>  
> Now it may be time to reconsider that: having one future object per
> running task may become a scalability issue. It may be worth to use
> callbacks instead. It should not be too hard to refactor current engine
> for that. Also, as TaskExecutorBase is an internal API, there should not
> be any compatibility issues.
>  
> Then, we can make task executor interface public and allow clients to
> provide their own task executors. It will be possible then for Mistral
> to implement its own task executor, or several, and share the
> executors between all the engine instances.
>  
> You can call it a plan;)
>  
> == Running flows ==
>  
> To run the flow TaskFlow client uses engine interface; also, there are
> few of helper functions provided for convenience:
>  
> http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base
> http://docs.openstack.org/developer/taskflow/engines.html#creating-engines
>  
> That is part of our public API, it is stable and good enough. Basically,
> I don't think this API needs any major change.
>  
> Maybe it worth to add function or method to schedule running flow
> without actually waiting for flow completion (at least, it was on my top
> secret TODO list for quite a long time).
>  
> == Engine internals ==
>  
> Each engine eats resources, like thread it runs on; using these
> resources to run one flow only is somewhat wasteful. Some work is
> already planned to address this situation (see e.g.
> https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread).
> Also, it might be good idea to implement different 'type' of engine to
> support 'lazy' model, as Joshua suggests.
>  
> But whatever should and will be done about it, I daresay all that work
> can be done without affecting API more then I described above.
>  
> --  
> WBR,
> Ivan A. Melnikov
>  
> ... tasks must flow ...
>  
>  
> On 02.04.2014 01:51, Dmitri Zimine wrote:
> > Even more responses inline :)
>  
> [...]
>  
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org (mailto:OpenStack-dev at lists.openstack.org)
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140404/20a20b58/attachment.html>