Open Stack

Cool, feel free to jump on IRC[1] if u need in-person details/questions or
other (consultation is free, haha).

[1] irc://chat.freenode.net/openstack-state-management

-Josh

-----Original Message-----
From: Eoghan Glynn <eglynn at redhat.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Tuesday, June 17, 2014 at 3:11 PM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [taskflow] Recommendations for the
granularity of tasks and their "stickiness" to workers

>
>Thanks to Joshua and Sandy for the clarifications on the finer
>points of taskflow features and usage.
>
>I'll consume the docco linked, and will be back to annoy ye with
>further questions :)
>
>Cheers,
>Eoghan
>
>
>> Back on the distributed subject (since this deserves a different email),
>> 
>> In the newest taskflow release (0.3.x) we have 2 mechanisms for
>> distribution outside of a process.
>> 
>> One is the job/jobboard[1] & conductor[2] concepts,
>> 
>> These concepts allow for atomic ownership of a 'job' and conductors act
>>as
>> one way (not the only way) to perform the work in a distributed manner
>> (conductors consume and perform work). Conductors are pretty new so if
>> bugs are found please let us know. Jobs have some unique properties
>>that I
>> think make them attractive[3].
>> 
>> The second method here is what is called the W.B.E. (the worker based
>> engine),
>> 
>> This is different in that it is a distribution at the engine[4] layer
>>(not
>> at the job layer), this engine allows for running tasks on remote
>>workers
>> (and it can be used in combination with the job concept/layer).
>> Documentation for this can be found @
>> http://docs.openstack.org/developer/taskflow/workers.html; this WBE is
>> under development, it does work, but it does have a few limitations that
>> still need addressing (see docs link).
>> 
>> So that¹s what exists currently (not just in-process things),
>> 
>> [1] http://docs.openstack.org/developer/taskflow/jobs.html
>> [2] http://docs.openstack.org/developer/taskflow/conductors.html
>> [3] http://docs.openstack.org/developer/taskflow/jobs.html#features
>> [4] http://docs.openstack.org/developer/taskflow/engines.html
>> 
>> -----Original Message-----
>> From: Sandy Walsh <sandy.walsh at RACKSPACE.COM>
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> <openstack-dev at lists.openstack.org>
>> Date: Tuesday, June 17, 2014 at 5:33 AM
>> To: "OpenStack Development Mailing List (not for usage questions)"
>> <openstack-dev at lists.openstack.org>
>> Subject: Re: [openstack-dev] [taskflow] Recommendations for the
>> granularity of tasks and their "stickiness" to workers
>> 
>> >On 6/17/2014 7:04 AM, Eoghan Glynn wrote:
>> >> Folks,
>> >>
>> >> A question for the taskflow ninjas.
>> >>
>> >> Any thoughts on best practice WRT $subject?
>> >>
>> >> Specifically I have in mind this ceilometer review[1] which adopts
>> >> the approach of using very fine-grained tasks (at the level of an
>> >> individual alarm evaluation) combined with short-term assignments
>> >> to individual workers.
>> >>
>> >> But I'm also thinking of future potential usage of taskflow within
>> >> ceilometer, to support partitioning of work over a scaled-out array
>> >> of central agents.
>> >>
>> >> Does taskflow also naturally support a model whereby more chunky
>> >> tasks (possibly including ongoing periodic work) are assigned to
>> >> workers in a stickier fashion, such that re-balancing of workload
>> >> can easily be triggered when a change is detected in the pool of
>> >> available workers?
>> >
>> >I don't think taskflow today is really focused on load balancing of
>> >tasks. Something like gearman [1] might be better suited in the near
>>term?
>> >
>> >My understanding is that taskflow is really focused on in-process tasks
>> >(with retry, restart, etc) and later will support distributed tasks.
>>But
>> >my data could be stale too. (jharlow?)
>> >
>> >Even still, the decision of smaller tasks vs. chunky ones really comes
>> >down to how much work you want to re-do if there is a failure. I've
>>seen
>> >some uses of taskflow where the breakdown of tasks seemed artificially
>> >small. Meaning, the overhead of going back to the library on an
>> >undo/rewind is greater than the undo itself.
>> >
>> >-S
>> >
>> >[1] http://gearman.org/
>> >
>> >
>> >_______________________________________________
>> >OpenStack-dev mailing list
>> >OpenStack-dev at lists.openstack.org
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev