[openstack-dev] [taskflow] Recommendations for the granularity of tasks and their "stickiness" to workers

Sandy Walsh sandy.walsh at RACKSPACE.COM
Tue Jun 17 12:33:42 UTC 2014

On 6/17/2014 7:04 AM, Eoghan Glynn wrote:
> Folks,
> A question for the taskflow ninjas.
> Any thoughts on best practice WRT $subject?
> Specifically I have in mind this ceilometer review[1] which adopts
> the approach of using very fine-grained tasks (at the level of an
> individual alarm evaluation) combined with short-term assignments
> to individual workers.
> But I'm also thinking of future potential usage of taskflow within
> ceilometer, to support partitioning of work over a scaled-out array
> of central agents.
> Does taskflow also naturally support a model whereby more chunky
> tasks (possibly including ongoing periodic work) are assigned to
> workers in a stickier fashion, such that re-balancing of workload
> can easily be triggered when a change is detected in the pool of
> available workers?

I don't think taskflow today is really focused on load balancing of
tasks. Something like gearman [1] might be better suited in the near term?

My understanding is that taskflow is really focused on in-process tasks
(with retry, restart, etc) and later will support distributed tasks. But
my data could be stale too. (jharlow?)

Even still, the decision of smaller tasks vs. chunky ones really comes
down to how much work you want to re-do if there is a failure. I've seen
some uses of taskflow where the breakdown of tasks seemed artificially
small. Meaning, the overhead of going back to the library on an
undo/rewind is greater than the undo itself.


[1] http://gearman.org/

More information about the OpenStack-dev mailing list