<div dir="ltr"><div><span style="font-size:12.8px">Hi Joshua,</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks for all your inputs.</span></div><div><span style="font-size:12.8px">We are using this feature successfully. But I rarely see an issue related to concurrency.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">To give you a brief, w</span><span style="font-size:12.8px">e use eventlets and every job runs in a separate eventlet thread. </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In the job execution part, we use taskflow functionality and persist all the details of the job.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">There we try to get the backend as below. </span>We do upgrade everytime when a job is executed.</div><div><br></div><div><div style=""><div>        backend_uri = cfg.CONF.db.sqlalchemy_url</div><div>        conf = {</div><div>            'connection': backend_uri,</div><div>        }</div><div>        self.backend = backends.fetch(conf)</div><div>        with contextlib.closing(self.backend.get_connection()) as conn:<br></div><div>            conn.upgrade()</div><div><br></div><div>Now when two jobs start executing at the same time, I see below error.</div><div><div>Failed upgrading database version</div><div>  DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=14></div></div><div><br></div><div>We have monkey patched eventlet, is it not supposed to take care of these concurrency issues?</div><div><br></div><div>Below are the versions for related modules in use:</div><div>eventlet==0.17.4<br></div><div>taskflow==1.26.0<br></div><div>SQLAlchemy==1.0.9<br></div><div><br></div><div>Thanks,</div><div>Kanthi</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 12, 2016 at 1:29 PM, Joshua Harlow <span dir="ltr"><<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">pn kk wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Joshua,<br>
<br>
Yes, sure will do that once I get some window out of my work.<br>
<br>
One last query(hopefully :) ) , can the factory method be an instance<br>
method of a class?<br>
</blockquote>
<br></span>
Instance methods are particularly hard to use (since they require an instance of an object to be useful); so I think the check u have hit is making sure that if the flow-factory is called to recreate the flow that it can do so without having import issues. Currently I believe it doesn't handle instance methods (due to the complexity of needing an instance attached to that method).<br>
<br>
Perhaps what u want is to provide a function that can be found like:<br>
<br>
def make_flow_factory():<br>
   return FlowFactory().flow_factory<br>
<br>
Or make flow_factory a class or static method, which should have the same/similar effect.<br>
<br>
Hope that makes sense (more queries are fine and welcome!)<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
I tried giving it as "FlowFactory().flow_factory", where FlowFactory is<br>
my class name. It failed with below error:<br>
ValueError: Flow factory <bound method FlowFactory.flow_factory of<br>
<__main__.FlowFactory instance at 0x2b43b48>> is not reimportable by<br>
name __builtin__.instance.flow_factory<br>
<br>
Thanks again<br>
-Kanthi<br>
<br>
<br>
On Thu, Jan 28, 2016 at 12:29 AM, Joshua Harlow <<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a><br></span><div><div class="h5">
<mailto:<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>>> wrote:<br>
<br>
    pn kk wrote:<br>
<br>
        Hi,<br>
<br>
        Thanks for the responses. Putting it in a small example<br>
<br>
        def flow_factory(tmp):<br>
              return lf.Flow('resume from backend example').add(<br>
                  TestTask(name='first', test=tmp),<br>
                  InterruptTask(name='boom'),<br>
                  TestTask(name='second', test="second task"))<br>
<br>
<br>
        class TestTask(task.Task):<br>
              def __init__(self, name, provides=None, test, **kwargs):<br>
                  self.test=test<br>
                  super(TestTask, self).__init__(name, provides, **kwargs)<br>
              def execute(self, *args, **kwargs):<br>
                  print('executing %s' % self)<br>
                  return 'ok'<br>
<br>
        class InterruptTask(task.Task):<br>
              def execute(self, *args, **kwargs):<br>
                  # DO NOT TRY THIS AT HOME<br>
                  engine.suspend()<br>
<br>
        I was searching for a way in which I can reload the flow after crash<br>
        without passing the parameter "tmp" shown above<br>
        Looks like "load_from_factory" gives this provision.<br>
<br>
<br>
    Thanks for the example, ya, this is one such way to do this that u<br>
    have found, there are a few other ways, but that is the main one<br>
    that people should be using.<br>
<br>
<br>
<br>
        engine =<br>
        taskflow.engines.load_from_factory(flow_factory=flow_factory,<br>
        factory_kwargs={"tmp":"test_data"}, book=book, backend=backend)<br>
        engine.run()<br>
<br>
        Now it suspends after running interrupt task, I can now reload<br>
        the flow<br>
        from the saved factory method without passing parameter again.<br>
        for flow_detail_2 in book:<br>
              engine2 = taskflow.engines.load_from_detail(flow_detail_2,<br>
        backend=backend)<br>
<br>
        engine2.run()<br>
<br>
        Let me know if this is ok or is there a better approach to<br>
        achieve this?<br>
<br>
<br>
    There are a few other ways, but this one should be the currently<br>
    recommended one.<br>
<br>
    An idea, do u want to maybe update (submit a review to update) the<br>
    docs, if u didn't find this very easy to figure out so that others<br>
    can more easily figure it out. I'm sure that would be appreciated by<br>
    all.<br>
<br>
<br>
        -Thanks<br>
<br>
<br>
        On Wed, Jan 27, 2016 at 12:03 AM, Joshua Harlow<br>
        <<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a> <mailto:<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>><br></div></div>
        <mailto:<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a> <mailto:<a href="mailto:harlowja@fastmail.com" target="_blank">harlowja@fastmail.com</a>>>><div><div class="h5"><br>
        wrote:<br>
<br>
             Hi there,<br>
<br>
             Michał is correct, it should be saved.<br>
<br>
             Do u have a small example of what u are trying to do<br>
        because that<br>
             will help determine if what u are doing will be saved or<br>
        whether it<br>
             will not be.<br>
<br>
             Or even possibly explaining what is being done would be fine to<br>
             (more data/info for me to reason about what should be<br>
        stored in your<br>
             case).<br>
<br>
             Thanks,<br>
<br>
             Josh<br>
<br>
<br>
             Michał Dulko wrote:<br>
<br>
                 On 01/26/2016 10:23 AM, pn kk wrote:<br>
<br>
                     Hi,<br>
<br>
                     I use taskflow for job management and now trying to<br>
        persist<br>
                     the state<br>
                     of flows/tasks in mysql to recover incase of<br>
        process crashes.<br>
<br>
                     I could see the state and the task results stored<br>
        in the<br>
                     database.<br>
<br>
                     Now I am looking for some way to store the input<br>
        parameters<br>
                     of the tasks.<br>
<br>
                     Please share your inputs to achieve this.<br>
<br>
                     -Thanks<br>
<br>
                 I've played with that some time ago and if I recall<br>
        correctly input<br>
                 parameters should be available in the flow's storage, which<br>
                 means these<br>
                 are also saved to the DB. Take a look on<br>
        resume_workflows method<br>
                 on my<br>
                 old PoC [1] (hopefully TaskFlow haven't changed much<br>
        since then).<br>
<br>
                 [1]<br>
        <a href="https://review.openstack.org/#/c/152200/4/cinder/scheduler/manager.py" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/152200/4/cinder/scheduler/manager.py</a><br>
<br>
<br>
        __________________________________________________________________________<br>
                 OpenStack Development Mailing List (not for usage<br>
        questions)<br>
                 Unsubscribe:<br>
        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
        __________________________________________________________________________<br>
             OpenStack Development Mailing List (not for usage questions)<br>
             Unsubscribe:<br>
        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
        __________________________________________________________________________<br>
        OpenStack Development Mailing List (not for usage questions)<br>
        Unsubscribe:<br>
        <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
        <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
        <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
    __________________________________________________________________________<br>
    OpenStack Development Mailing List (not for usage questions)<br>
    Unsubscribe:<br>
    <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
    <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
    <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote><div class="HOEnZb"><div class="h5">
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>