[openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

Joshua Harlow harlowja at fastmail.com
Sat Jan 9 00:54:05 UTC 2016


ESWAR RAO wrote:
> Hi Joshua,
>
> Thanks for the response.
>
> As you mentioned, I went through below which has taskflow as celery
> front end:
>
> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>
> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery

I think the idea way-back-when was celery would be a backend, not a 
front-end, although my memory might be fuzzy in this area (since it was 
a few years ago).

>
>
> I guess if we have a job = task1 + task2 ; if we execute them through a
> taskflow parallel pattern, they will be executed in parallel.
>
> Just curious to know that since this threading is built on
> eventlets/green threads/python threads they will be affected by GIL and
> we may not utilize multi-core capability of our systems.

So that assumes u are using a taskflow engine which is powered by 
threads (or greenthreads), there are two types that do not use threads 
but actually run out of process:

http://docs.openstack.org/developer/taskflow/engines.html#types

1. http://docs.openstack.org/developer/taskflow/workers.html (runs out 
of process, across different machines, using kombu for triggering 
start/status... of tasks on other machines)

2. A parallel engine that uses a process pool executor 
(https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor) 
will then execute tasks using out of process child-processes.

So both of these don't have the same GIL issue (although greenthreads by 
there very nature are not affected by the GIL either).

The example (aptly named hello world) shows these different types 
(although it doesn't show the worker based one, since that requires more 
things to setup) @ 
https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82

-Josh

>
> It seems in celery, we can have these tasks run as multiple processes so
> that they are not affected by GIL.

Right, the above starts to show why taskflow is really a super-set of 
celery (for some combination/interpretation of what taskflow or celery do).

>
> Please correct me if my understanding is wrong ??
>
> Thanks
> Eswar
>
>
> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlowja at fastmail.com
> <mailto:harlowja at fastmail.com>> wrote:
>
>     I also updated
>     https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>     that said wiki is no longer active (it was an attempt to back a
>     taskflow engine[1] with celery); although if u are interested in
>     continuing down this path feel free.
>
>     Hopefully that clears up some 'confusion' around that wiki.
>
>     [1] http://docs.openstack.org/developer/taskflow/engines.html
>
>
>     Joshua Harlow wrote:
>
>         So actually they are quite different, (although similar at some
>         level),
>
>         Given that celery isn't really a replacement for taskflow
>         although one
>         could say, from what I've heard from others, that taskflow is a
>         super-set of what celery is so taskflow likely can replace parts of
>         celery (but not vice-versa).
>
>         Feel free to jump on #openstack-state-management IRC channel if
>         u want
>         to chat in person more about why (it gets into details that
>         might just
>         be easier to explain in person).
>
>         ESWAR RAO wrote:
>
>             Hi All,
>
>             Please let me know whether celery is replacement for taskflow.
>
>             As per my understanding, task-flow can break jobs into tasks
>             and execute
>             them.
>
>              From celery wiki, it also does almost similar behaviour.
>
>             I guess in most of openstack components taskflow is widely used.
>             Any places where its being replaced with celery ??
>
>             Celery: https://wiki.openstack.org/wiki/Celery
>             Distributed:
>             https://wiki.openstack.org/wiki/DistributedTaskManagement
>             TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>
>             Thanks
>             Eswar
>
>             __________________________________________________________________________
>
>             OpenStack Development Mailing List (not for usage questions)
>             Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>         _______________________________________________
>         Mailing list:
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>         Post to : openstack at lists.openstack.org
>         <mailto:openstack at lists.openstack.org>
>         Unsubscribe :
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>     _______________________________________________
>     Mailing list:
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>     Post to     : openstack at lists.openstack.org
>     <mailto:openstack at lists.openstack.org>
>     Unsubscribe :
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>



More information about the OpenStack-dev mailing list