Open Stack

Hi, Eswar,

In case you're interested, there is another choice of 'workflow
engine' you can refer to: http://docs.openstack.org/developer/mistral/

Hope you'll find it interesting :-)

On Sat, Jan 9, 2016 at 10:33 AM, ESWAR RAO <eswar7028 at gmail.com> wrote:
> Thanks Joshua for the clear explanation.
>
> On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow <harlowja at fastmail.com> wrote:
>>
>> ESWAR RAO wrote:
>>>
>>> Hi Joshua,
>>>
>>> Thanks for the response.
>>>
>>> As you mentioned, I went through below which has taskflow as celery
>>> front end:
>>>
>>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>>
>>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>>
>> I think the idea way-back-when was celery would be a backend, not a
>> front-end, although my memory might be fuzzy in this area (since it was a
>> few years ago).
>>
>>>
>>>
>>> I guess if we have a job = task1 + task2 ; if we execute them through a
>>> taskflow parallel pattern, they will be executed in parallel.
>>>
>>> Just curious to know that since this threading is built on
>>> eventlets/green threads/python threads they will be affected by GIL and
>>> we may not utilize multi-core capability of our systems.
>>
>>
>> So that assumes u are using a taskflow engine which is powered by threads
>> (or greenthreads), there are two types that do not use threads but actually
>> run out of process:
>>
>> http://docs.openstack.org/developer/taskflow/engines.html#types
>>
>> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
>> process, across different machines, using kombu for triggering
>> start/status... of tasks on other machines)
>>
>> 2. A parallel engine that uses a process pool executor
>> (https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
>> will then execute tasks using out of process child-processes.
>>
>> So both of these don't have the same GIL issue (although greenthreads by
>> there very nature are not affected by the GIL either).
>>
>> The example (aptly named hello world) shows these different types
>> (although it doesn't show the worker based one, since that requires more
>> things to setup) @
>> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>>
>> -Josh
>>
>>>
>>> It seems in celery, we can have these tasks run as multiple processes so
>>> that they are not affected by GIL.
>>
>>
>> Right, the above starts to show why taskflow is really a super-set of
>> celery (for some combination/interpretation of what taskflow or celery do).
>>
>>>
>>> Please correct me if my understanding is wrong ??
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlowja at fastmail.com
>>> <mailto:harlowja at fastmail.com>> wrote:
>>>
>>>     I also updated
>>>     https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>>>     that said wiki is no longer active (it was an attempt to back a
>>>     taskflow engine[1] with celery); although if u are interested in
>>>     continuing down this path feel free.
>>>
>>>     Hopefully that clears up some 'confusion' around that wiki.
>>>
>>>     [1] http://docs.openstack.org/developer/taskflow/engines.html
>>>
>>>
>>>     Joshua Harlow wrote:
>>>
>>>         So actually they are quite different, (although similar at some
>>>         level),
>>>
>>>         Given that celery isn't really a replacement for taskflow
>>>         although one
>>>         could say, from what I've heard from others, that taskflow is a
>>>         super-set of what celery is so taskflow likely can replace parts
>>> of
>>>         celery (but not vice-versa).
>>>
>>>         Feel free to jump on #openstack-state-management IRC channel if
>>>         u want
>>>         to chat in person more about why (it gets into details that
>>>         might just
>>>         be easier to explain in person).
>>>
>>>         ESWAR RAO wrote:
>>>
>>>             Hi All,
>>>
>>>             Please let me know whether celery is replacement for
>>> taskflow.
>>>
>>>             As per my understanding, task-flow can break jobs into tasks
>>>             and execute
>>>             them.
>>>
>>>              From celery wiki, it also does almost similar behaviour.
>>>
>>>             I guess in most of openstack components taskflow is widely
>>> used.
>>>             Any places where its being replaced with celery ??
>>>
>>>             Celery: https://wiki.openstack.org/wiki/Celery
>>>             Distributed:
>>>             https://wiki.openstack.org/wiki/DistributedTaskManagement
>>>             TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>>
>>>             Thanks
>>>             Eswar
>>>
>>>
>>> __________________________________________________________________________
>>>
>>>             OpenStack Development Mailing List (not for usage questions)
>>>             Unsubscribe:
>>>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>         _______________________________________________
>>>         Mailing list:
>>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>         Post to : openstack at lists.openstack.org
>>>         <mailto:openstack at lists.openstack.org>
>>>         Unsubscribe :
>>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>>     _______________________________________________
>>>     Mailing list:
>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>     Post to     : openstack at lists.openstack.org
>>>     <mailto:openstack at lists.openstack.org>
>>>     Unsubscribe :
>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards!
-----------------------------------
Lingxian Kong