Open Stack

Thanks to Joshua and Sandy for the clarifications on the finer
points of taskflow features and usage.

I'll consume the docco linked, and will be back to annoy ye with
further questions :)

Cheers,
Eoghan


> Back on the distributed subject (since this deserves a different email),
> 
> In the newest taskflow release (0.3.x) we have 2 mechanisms for
> distribution outside of a process.
> 
> One is the job/jobboard[1] & conductor[2] concepts,
> 
> These concepts allow for atomic ownership of a 'job' and conductors act as
> one way (not the only way) to perform the work in a distributed manner
> (conductors consume and perform work). Conductors are pretty new so if
> bugs are found please let us know. Jobs have some unique properties that I
> think make them attractive[3].
> 
> The second method here is what is called the W.B.E. (the worker based
> engine),
> 
> This is different in that it is a distribution at the engine[4] layer (not
> at the job layer), this engine allows for running tasks on remote workers
> (and it can be used in combination with the job concept/layer).
> Documentation for this can be found @
> http://docs.openstack.org/developer/taskflow/workers.html; this WBE is
> under development, it does work, but it does have a few limitations that
> still need addressing (see docs link).
> 
> So that¹s what exists currently (not just in-process things),
> 
> [1] http://docs.openstack.org/developer/taskflow/jobs.html
> [2] http://docs.openstack.org/developer/taskflow/conductors.html
> [3] http://docs.openstack.org/developer/taskflow/jobs.html#features
> [4] http://docs.openstack.org/developer/taskflow/engines.html
> 
> -----Original Message-----
> From: Sandy Walsh <sandy.walsh at RACKSPACE.COM>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>
> Date: Tuesday, June 17, 2014 at 5:33 AM
> To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [taskflow] Recommendations for the
> granularity of tasks and their "stickiness" to workers
> 
> >On 6/17/2014 7:04 AM, Eoghan Glynn wrote:
> >> Folks,
> >>
> >> A question for the taskflow ninjas.
> >>
> >> Any thoughts on best practice WRT $subject?
> >>
> >> Specifically I have in mind this ceilometer review[1] which adopts
> >> the approach of using very fine-grained tasks (at the level of an
> >> individual alarm evaluation) combined with short-term assignments
> >> to individual workers.
> >>
> >> But I'm also thinking of future potential usage of taskflow within
> >> ceilometer, to support partitioning of work over a scaled-out array
> >> of central agents.
> >>
> >> Does taskflow also naturally support a model whereby more chunky
> >> tasks (possibly including ongoing periodic work) are assigned to
> >> workers in a stickier fashion, such that re-balancing of workload
> >> can easily be triggered when a change is detected in the pool of
> >> available workers?
> >
> >I don't think taskflow today is really focused on load balancing of
> >tasks. Something like gearman [1] might be better suited in the near term?
> >
> >My understanding is that taskflow is really focused on in-process tasks
> >(with retry, restart, etc) and later will support distributed tasks. But
> >my data could be stale too. (jharlow?)
> >
> >Even still, the decision of smaller tasks vs. chunky ones really comes
> >down to how much work you want to re-do if there is a failure. I've seen
> >some uses of taskflow where the breakdown of tasks seemed artificially
> >small. Meaning, the overhead of going back to the library on an
> >undo/rewind is greater than the undo itself.
> >
> >-S
> >
> >[1] http://gearman.org/
> >
> >
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>