Open Stack

Back on the distributed subject (since this deserves a different email),

In the newest taskflow release (0.3.x) we have 2 mechanisms for
distribution outside of a process.

One is the job/jobboard[1] & conductor[2] concepts,

These concepts allow for atomic ownership of a 'job' and conductors act as
one way (not the only way) to perform the work in a distributed manner
(conductors consume and perform work). Conductors are pretty new so if
bugs are found please let us know. Jobs have some unique properties that I
think make them attractive[3].

The second method here is what is called the W.B.E. (the worker based
engine),

This is different in that it is a distribution at the engine[4] layer (not
at the job layer), this engine allows for running tasks on remote workers
(and it can be used in combination with the job concept/layer).
Documentation for this can be found @
http://docs.openstack.org/developer/taskflow/workers.html; this WBE is
under development, it does work, but it does have a few limitations that
still need addressing (see docs link).

So that¹s what exists currently (not just in-process things),

[1] http://docs.openstack.org/developer/taskflow/jobs.html
[2] http://docs.openstack.org/developer/taskflow/conductors.html
[3] http://docs.openstack.org/developer/taskflow/jobs.html#features
[4] http://docs.openstack.org/developer/taskflow/engines.html

-----Original Message-----
From: Sandy Walsh <sandy.walsh at RACKSPACE.COM>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Tuesday, June 17, 2014 at 5:33 AM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [taskflow] Recommendations for the
granularity of tasks and their "stickiness" to workers

>On 6/17/2014 7:04 AM, Eoghan Glynn wrote:
>> Folks,
>>
>> A question for the taskflow ninjas.
>>
>> Any thoughts on best practice WRT $subject?
>>
>> Specifically I have in mind this ceilometer review[1] which adopts
>> the approach of using very fine-grained tasks (at the level of an
>> individual alarm evaluation) combined with short-term assignments
>> to individual workers.
>>
>> But I'm also thinking of future potential usage of taskflow within
>> ceilometer, to support partitioning of work over a scaled-out array
>> of central agents.
>>
>> Does taskflow also naturally support a model whereby more chunky
>> tasks (possibly including ongoing periodic work) are assigned to
>> workers in a stickier fashion, such that re-balancing of workload
>> can easily be triggered when a change is detected in the pool of
>> available workers?
>
>I don't think taskflow today is really focused on load balancing of
>tasks. Something like gearman [1] might be better suited in the near term?
>
>My understanding is that taskflow is really focused on in-process tasks
>(with retry, restart, etc) and later will support distributed tasks. But
>my data could be stale too. (jharlow?)
>
>Even still, the decision of smaller tasks vs. chunky ones really comes
>down to how much work you want to re-do if there is a failure. I've seen
>some uses of taskflow where the breakdown of tasks seemed artificially
>small. Meaning, the overhead of going back to the library on an
>undo/rewind is greater than the undo itself.
>
>-S
>
>[1] http://gearman.org/
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev