Hi Sachin, I'm not 100% sure I understand your need, but I will attempt to answer and you can correct me if I am off base. Taskflow engines (you can create as many of these as you want) use an executor defined at flow load time. Here is a snippet from the Octavia code: self.executor = concurrent.futures.ThreadPoolExecutor( max_workers=CONF.task_flow.max_workers) eng = tf_engines.load( flow, engine=CONF.task_flow.engine, executor=self.executor, never_resolve=CONF.task_flow.disable_revert, **kwargs) The parts you are likely interested in are: 1. The executor. In this case we are using a concurrent.futures.ThreadPoolExecutor. We then set the max_workers setting to the number of threads we want in our taskflow engine thread pool. 2. During flow load, we define the engine to be 'parallel' (note: 'serial' is the default). This means that unordered flows will run in parallel as opposed to serially. 3. As noted in the documentation[1], You can share an executor between taskflow engines to share the thread pool. Finally, you want to use "unordered" flows or sub-flows to execute tasks concurrently. [1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel Michael On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha@gmail.com> wrote:
Hi,
We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool.
Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread.
Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel.
I came across iter_run method of the engine class. But not sure if that can be used for this purpose.
Any help is highly appreciated.