Open Stack

I'm opinionated because I work with/on Taskflow and had mostly bad
experiences with Celery, but here's my $0.02.  It's possible that the
codebase I inherited just made bad use of Celery or things have improved a
lot in the last 18 months, but all I can speak from is my own experience.

Taskflow and Celery at a high level can be used for the same thing, but if
anything Taskflow would be considered a replacement for Celery, IMO.  We
migrated a codebase from Celery to Taskflow because Taskflow gives you a
lot more capabilities related to managing the whole process rather than
just a bunch of individual async tasks.  Some still swear by Celery
because they like that they can just throw some decorators on existing
methods and execute them asynchronously, and if that's all you need,
Taskflow is probably overkill.  For us, we wanted a lot more control over
the whole flow, including how to handle task failures, being able to run
many things in parallel and others in serial, etc. Just conceptually
thinking of the process as a series of tasks in a flow makes it a lot
easier to reason about what is going on than having to trace throw a bunch
of asynchronous functions. We also liked the resiliency guarantees that
Taskflow's job board and conductor/worker engine provided more than what
Celery had at the time (it didn't seem to guarantee that tasks were
executed, and we had fairly often where rabbitmq connections would time
out and our process would just not continue, no log, no error, just the
next task was never executed).

Greg

On 1/8/16, 3:19 PM, "Joshua Harlow" <harlowja at fastmail.com> wrote:

>So actually they are quite different, (although similar at some level),
>
>Given that celery isn't really a replacement for taskflow although one
>could say, from what I've heard from others, that taskflow is a
>super-set of what celery is so taskflow likely can replace parts of
>celery (but not vice-versa).
>
>Feel free to jump on #openstack-state-management IRC channel if u want
>to chat in person more about why (it gets into details that might just
>be easier to explain in person).
>
>ESWAR RAO wrote:
>> Hi All,
>>
>> Please let me know whether celery is replacement for taskflow.
>>
>> As per my understanding, task-flow can break jobs into tasks and execute
>> them.
>>
>>  From celery wiki, it also does almost similar behaviour.
>>
>> I guess in most of openstack components taskflow is widely used.
>> Any places where its being replaced with celery ??
>>
>> Celery: https://wiki.openstack.org/wiki/Celery
>> Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>
>> Thanks
>> Eswar
>>
>> 
>>_________________________________________________________________________
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev