[TaskFlow] running multiple engines on a shared thread
Hi, We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool. Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread. Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel. I came across iter_run method of the engine class. But not sure if that can be used for this purpose. Any help is highly appreciated.
Hi Sachin, I'm not 100% sure I understand your need, but I will attempt to answer and you can correct me if I am off base. Taskflow engines (you can create as many of these as you want) use an executor defined at flow load time. Here is a snippet from the Octavia code: self.executor = concurrent.futures.ThreadPoolExecutor( max_workers=CONF.task_flow.max_workers) eng = tf_engines.load( flow, engine=CONF.task_flow.engine, executor=self.executor, never_resolve=CONF.task_flow.disable_revert, **kwargs) The parts you are likely interested in are: 1. The executor. In this case we are using a concurrent.futures.ThreadPoolExecutor. We then set the max_workers setting to the number of threads we want in our taskflow engine thread pool. 2. During flow load, we define the engine to be 'parallel' (note: 'serial' is the default). This means that unordered flows will run in parallel as opposed to serially. 3. As noted in the documentation[1], You can share an executor between taskflow engines to share the thread pool. Finally, you want to use "unordered" flows or sub-flows to execute tasks concurrently. [1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel Michael On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha@gmail.com> wrote:
Hi,
We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool.
Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread.
Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel.
I came across iter_run method of the engine class. But not sure if that can be used for this purpose.
Any help is highly appreciated.
thanks Michael, I am aware that executor can be reused by different engines. My query was regarding if multiple engines can share same thread for running the engines(and not the tasks of those engines). I tried run_iter which can be used to run multiple engines but the tasks of individual engines are run one after another. Probably engine is holding on to the thread. This is limiting our ability run multiple workflows (i.e. engines) in parallel. My query is - is it possible to run multiple engines in parallel on same thread (using some asynchronous task execution) On Thu, Feb 27, 2020 at 3:18 AM Michael Johnson <johnsomor@gmail.com> wrote:
Hi Sachin,
I'm not 100% sure I understand your need, but I will attempt to answer and you can correct me if I am off base.
Taskflow engines (you can create as many of these as you want) use an executor defined at flow load time.
Here is a snippet from the Octavia code: self.executor = concurrent.futures.ThreadPoolExecutor( max_workers=CONF.task_flow.max_workers) eng = tf_engines.load( flow, engine=CONF.task_flow.engine, executor=self.executor, never_resolve=CONF.task_flow.disable_revert, **kwargs)
The parts you are likely interested in are:
1. The executor. In this case we are using a concurrent.futures.ThreadPoolExecutor. We then set the max_workers setting to the number of threads we want in our taskflow engine thread pool. 2. During flow load, we define the engine to be 'parallel' (note: 'serial' is the default). This means that unordered flows will run in parallel as opposed to serially. 3. As noted in the documentation[1], You can share an executor between taskflow engines to share the thread pool.
Finally, you want to use "unordered" flows or sub-flows to execute tasks concurrently.
[1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel
Michael
On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha@gmail.com> wrote:
Hi,
We are using taskflow to execute workflows. Each workflow is executed by
a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool.
Most of the time, the workflow tasks are run by agents which could take
some time to complete. Each engine is alive and runs on a dedicated thread.
Is there any way to reuse or run multiple engines on one thread. The
individual tasks of these engines can run in parallel.
I came across iter_run method of the engine class. But not sure if that
can be used for this purpose.
Any help is highly appreciated.
Hi Sachin, Ok, I think I understand what you are trying better now. Do you have a link to your code for reference? The parallelism of tasks inside a flow are dictated by three things: 1. Are you using the parallel action engine? (i.e. setting engine='parallel' when loading) 2. Is the flow of type "unordered flow". 3. The executor defined for the parallel engine. Flows are graphs, so if the flow is defined as a linear flow, it will run each task in a serial manner. Likewise, if the engine is set to "serial" (the default), even the unordered flows will run sequentially. Michael On Thu, Feb 27, 2020 at 1:19 AM Sachin Laddha <sachinladdha@gmail.com> wrote:
thanks Michael,
I am aware that executor can be reused by different engines. My query was regarding if multiple engines can share same thread for running the engines(and not the tasks of those engines).
I tried run_iter which can be used to run multiple engines but the tasks of individual engines are run one after another. Probably engine is holding on to the thread.
This is limiting our ability run multiple workflows (i.e. engines) in parallel.
My query is - is it possible to run multiple engines in parallel on same thread (using some asynchronous task execution)
On Thu, Feb 27, 2020 at 3:18 AM Michael Johnson <johnsomor@gmail.com> wrote:
Hi Sachin,
I'm not 100% sure I understand your need, but I will attempt to answer and you can correct me if I am off base.
Taskflow engines (you can create as many of these as you want) use an executor defined at flow load time.
Here is a snippet from the Octavia code: self.executor = concurrent.futures.ThreadPoolExecutor( max_workers=CONF.task_flow.max_workers) eng = tf_engines.load( flow, engine=CONF.task_flow.engine, executor=self.executor, never_resolve=CONF.task_flow.disable_revert, **kwargs)
The parts you are likely interested in are:
1. The executor. In this case we are using a concurrent.futures.ThreadPoolExecutor. We then set the max_workers setting to the number of threads we want in our taskflow engine thread pool. 2. During flow load, we define the engine to be 'parallel' (note: 'serial' is the default). This means that unordered flows will run in parallel as opposed to serially. 3. As noted in the documentation[1], You can share an executor between taskflow engines to share the thread pool.
Finally, you want to use "unordered" flows or sub-flows to execute tasks concurrently.
[1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel
Michael
On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha@gmail.com> wrote:
Hi,
We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool.
Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread.
Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel.
I came across iter_run method of the engine class. But not sure if that can be used for this purpose.
Any help is highly appreciated.
participants (2)
-
Michael Johnson
-
Sachin Laddha