Open Stack

Thanks Joshua for the clear explanation.

On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow <harlowja at fastmail.com> wrote:

> ESWAR RAO wrote:
>
>> Hi Joshua,
>>
>> Thanks for the response.
>>
>> As you mentioned, I went through below which has taskflow as celery
>> front end:
>>
>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>
>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>
> I think the idea way-back-when was celery would be a backend, not a
> front-end, although my memory might be fuzzy in this area (since it was a
> few years ago).
>
>
>>
>> I guess if we have a job = task1 + task2 ; if we execute them through a
>> taskflow parallel pattern, they will be executed in parallel.
>>
>> Just curious to know that since this threading is built on
>> eventlets/green threads/python threads they will be affected by GIL and
>> we may not utilize multi-core capability of our systems.
>>
>
> So that assumes u are using a taskflow engine which is powered by threads
> (or greenthreads), there are two types that do not use threads but actually
> run out of process:
>
> http://docs.openstack.org/developer/taskflow/engines.html#types
>
> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
> process, across different machines, using kombu for triggering
> start/status... of tasks on other machines)
>
> 2. A parallel engine that uses a process pool executor (
> https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
> will then execute tasks using out of process child-processes.
>
> So both of these don't have the same GIL issue (although greenthreads by
> there very nature are not affected by the GIL either).
>
> The example (aptly named hello world) shows these different types
> (although it doesn't show the worker based one, since that requires more
> things to setup) @
> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>
> -Josh
>
>
>> It seems in celery, we can have these tasks run as multiple processes so
>> that they are not affected by GIL.
>>
>
> Right, the above starts to show why taskflow is really a super-set of
> celery (for some combination/interpretation of what taskflow or celery do).
>
>
>> Please correct me if my understanding is wrong ??
>>
>> Thanks
>> Eswar
>>
>>
>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlowja at fastmail.com
>> <mailto:harlowja at fastmail.com>> wrote:
>>
>>     I also updated
>>     https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>>     that said wiki is no longer active (it was an attempt to back a
>>     taskflow engine[1] with celery); although if u are interested in
>>     continuing down this path feel free.
>>
>>     Hopefully that clears up some 'confusion' around that wiki.
>>
>>     [1] http://docs.openstack.org/developer/taskflow/engines.html
>>
>>
>>     Joshua Harlow wrote:
>>
>>         So actually they are quite different, (although similar at some
>>         level),
>>
>>         Given that celery isn't really a replacement for taskflow
>>         although one
>>         could say, from what I've heard from others, that taskflow is a
>>         super-set of what celery is so taskflow likely can replace parts
>> of
>>         celery (but not vice-versa).
>>
>>         Feel free to jump on #openstack-state-management IRC channel if
>>         u want
>>         to chat in person more about why (it gets into details that
>>         might just
>>         be easier to explain in person).
>>
>>         ESWAR RAO wrote:
>>
>>             Hi All,
>>
>>             Please let me know whether celery is replacement for taskflow.
>>
>>             As per my understanding, task-flow can break jobs into tasks
>>             and execute
>>             them.
>>
>>              From celery wiki, it also does almost similar behaviour.
>>
>>             I guess in most of openstack components taskflow is widely
>> used.
>>             Any places where its being replaced with celery ??
>>
>>             Celery: https://wiki.openstack.org/wiki/Celery
>>             Distributed:
>>             https://wiki.openstack.org/wiki/DistributedTaskManagement
>>             TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>
>>             Thanks
>>             Eswar
>>
>>
>> __________________________________________________________________________
>>
>>             OpenStack Development Mailing List (not for usage questions)
>>             Unsubscribe:
>>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>         _______________________________________________
>>         Mailing list:
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>         Post to : openstack at lists.openstack.org
>>         <mailto:openstack at lists.openstack.org>
>>         Unsubscribe :
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>>     _______________________________________________
>>     Mailing list:
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     Post to     : openstack at lists.openstack.org
>>     <mailto:openstack at lists.openstack.org>
>>     Unsubscribe :
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160109/5820f118/attachment.html>