[openstack-dev] [TaskFlow] TaskFlow persistence

Joshua Harlow harlowja at fastmail.com
Wed Jan 27 18:59:34 UTC 2016


pn kk wrote:
> Hi,
>
> Thanks for the responses. Putting it in a small example
>
> def flow_factory(tmp):
>      return lf.Flow('resume from backend example').add(
>          TestTask(name='first', test=tmp),
>          InterruptTask(name='boom'),
>          TestTask(name='second', test="second task"))
>
>
> class TestTask(task.Task):
>      def __init__(self, name, provides=None, test, **kwargs):
>          self.test=test
>          super(TestTask, self).__init__(name, provides, **kwargs)
>      def execute(self, *args, **kwargs):
>          print('executing %s' % self)
>          return 'ok'
>
> class InterruptTask(task.Task):
>      def execute(self, *args, **kwargs):
>          # DO NOT TRY THIS AT HOME
>          engine.suspend()
>
> I was searching for a way in which I can reload the flow after crash
> without passing the parameter "tmp" shown above
> Looks like "load_from_factory" gives this provision.

Thanks for the example, ya, this is one such way to do this that u have 
found, there are a few other ways, but that is the main one that people 
should be using.

>
>
> engine = taskflow.engines.load_from_factory(flow_factory=flow_factory,
> factory_kwargs={"tmp":"test_data"}, book=book, backend=backend)
> engine.run()
>
> Now it suspends after running interrupt task, I can now reload the flow
> from the saved factory method without passing parameter again.
> for flow_detail_2 in book:
>      engine2 = taskflow.engines.load_from_detail(flow_detail_2,
> backend=backend)
>
> engine2.run()
>
> Let me know if this is ok or is there a better approach to achieve this?

There are a few other ways, but this one should be the currently 
recommended one.

An idea, do u want to maybe update (submit a review to update) the docs, 
if u didn't find this very easy to figure out so that others can more 
easily figure it out. I'm sure that would be appreciated by all.

>
> -Thanks
>
>
> On Wed, Jan 27, 2016 at 12:03 AM, Joshua Harlow <harlowja at fastmail.com
> <mailto:harlowja at fastmail.com>> wrote:
>
>     Hi there,
>
>     Michał is correct, it should be saved.
>
>     Do u have a small example of what u are trying to do because that
>     will help determine if what u are doing will be saved or whether it
>     will not be.
>
>     Or even possibly explaining what is being done would be fine to
>     (more data/info for me to reason about what should be stored in your
>     case).
>
>     Thanks,
>
>     Josh
>
>
>     Michał Dulko wrote:
>
>         On 01/26/2016 10:23 AM, pn kk wrote:
>
>             Hi,
>
>             I use taskflow for job management and now trying to persist
>             the state
>             of flows/tasks in mysql to recover incase of process crashes.
>
>             I could see the state and the task results stored in the
>             database.
>
>             Now I am looking for some way to store the input parameters
>             of the tasks.
>
>             Please share your inputs to achieve this.
>
>             -Thanks
>
>         I've played with that some time ago and if I recall correctly input
>         parameters should be available in the flow's storage, which
>         means these
>         are also saved to the DB. Take a look on resume_workflows method
>         on my
>         old PoC [1] (hopefully TaskFlow haven't changed much since then).
>
>         [1]
>         https://review.openstack.org/#/c/152200/4/cinder/scheduler/manager.py
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list