<div dir="ltr">Thanks <font color="#500050"><span style="font-size:12.8px">Joshua for the clear explanation.</span></font></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow <span dir="ltr"><<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">ESWAR RAO wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Joshua,<br>
<br>
Thanks for the response.<br>
<br>
As you mentioned, I went through below which has taskflow as celery<br>
front end:<br>
<br>
<a href="https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine</a><br>
<br>
<a href="https://blueprints.launchpad.net/taskflow/+spec/distributed-celery" rel="noreferrer" target="_blank">https://blueprints.launchpad.net/taskflow/+spec/distributed-celery</a><br>
</blockquote>
<br></span>
I think the idea way-back-when was celery would be a backend, not a front-end, although my memory might be fuzzy in this area (since it was a few years ago).<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
I guess if we have a job = task1 + task2 ; if we execute them through a<br>
taskflow parallel pattern, they will be executed in parallel.<br>
<br>
Just curious to know that since this threading is built on<br>
eventlets/green threads/python threads they will be affected by GIL and<br>
we may not utilize multi-core capability of our systems.<br>
</blockquote>
<br></span>
So that assumes u are using a taskflow engine which is powered by threads (or greenthreads), there are two types that do not use threads but actually run out of process:<br>
<br>
<a href="http://docs.openstack.org/developer/taskflow/engines.html#types" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/taskflow/engines.html#types</a><br>
<br>
1. <a href="http://docs.openstack.org/developer/taskflow/workers.html" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/taskflow/workers.html</a> (runs out of process, across different machines, using kombu for triggering start/status... of tasks on other machines)<br>
<br>
2. A parallel engine that uses a process pool executor (<a href="https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor" rel="noreferrer" target="_blank">https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor</a>) will then execute tasks using out of process child-processes.<br>
<br>
So both of these don't have the same GIL issue (although greenthreads by there very nature are not affected by the GIL either).<br>
<br>
The example (aptly named hello world) shows these different types (although it doesn't show the worker based one, since that requires more things to setup) @ <a href="https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82" rel="noreferrer" target="_blank">https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82</a><br>
<br>
-Josh<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It seems in celery, we can have these tasks run as multiple processes so<br>
that they are not affected by GIL.<br>
</blockquote>
<br></span>
Right, the above starts to show why taskflow is really a super-set of celery (for some combination/interpretation of what taskflow or celery do).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Please correct me if my understanding is wrong ??<br>
<br>
Thanks<br>
Eswar<br>
<br>
<br>
On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a><br></span><div><div class="h5">
<mailto:<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>>> wrote:<br>
<br>
I also updated<br>
<a href="https://wiki.openstack.org/wiki/DistributedTaskManagement" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/DistributedTaskManagement</a> to denote<br>
that said wiki is no longer active (it was an attempt to back a<br>
taskflow engine[1] with celery); although if u are interested in<br>
continuing down this path feel free.<br>
<br>
Hopefully that clears up some 'confusion' around that wiki.<br>
<br>
[1] <a href="http://docs.openstack.org/developer/taskflow/engines.html" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/taskflow/engines.html</a><br>
<br>
<br>
Joshua Harlow wrote:<br>
<br>
So actually they are quite different, (although similar at some<br>
level),<br>
<br>
Given that celery isn't really a replacement for taskflow<br>
although one<br>
could say, from what I've heard from others, that taskflow is a<br>
super-set of what celery is so taskflow likely can replace parts of<br>
celery (but not vice-versa).<br>
<br>
Feel free to jump on #openstack-state-management IRC channel if<br>
u want<br>
to chat in person more about why (it gets into details that<br>
might just<br>
be easier to explain in person).<br>
<br>
ESWAR RAO wrote:<br>
<br>
Hi All,<br>
<br>
Please let me know whether celery is replacement for taskflow.<br>
<br>
As per my understanding, task-flow can break jobs into tasks<br>
and execute<br>
them.<br>
<br>
From celery wiki, it also does almost similar behaviour.<br>
<br>
I guess in most of openstack components taskflow is widely used.<br>
Any places where its being replaced with celery ??<br>
<br>
Celery: <a href="https://wiki.openstack.org/wiki/Celery" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/Celery</a><br>
Distributed:<br>
<a href="https://wiki.openstack.org/wiki/DistributedTaskManagement" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/DistributedTaskManagement</a><br>
TaskFlow: <a href="https://wiki.openstack.org/wiki/TaskFlow" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/TaskFlow</a><br>
<br>
Thanks<br>
Eswar<br>
<br>
__________________________________________________________________________<br>
<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br></div></div>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class=""><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
_______________________________________________<br>
Mailing list:<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br></span>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>><span class=""><br>
Unsubscribe :<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
<br>
_______________________________________________<br>
Mailing list:<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br></span>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>><br>
Unsubscribe :<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
<br>
</blockquote>
</blockquote></div><br></div>