Open Stack

Hi Sachin,

I'm not 100% sure I understand your need, but I will attempt to answer
and you can correct me if I am off base.

Taskflow engines (you can create as many of these as you want) use an
executor defined at flow load time.

Here is a snippet from the Octavia code:
        self.executor = concurrent.futures.ThreadPoolExecutor(
            max_workers=CONF.task_flow.max_workers)
        eng = tf_engines.load(
            flow,
            engine=CONF.task_flow.engine,
            executor=self.executor,
            never_resolve=CONF.task_flow.disable_revert,
            **kwargs)

The parts you are likely interested in are:

1. The executor. In this case we are using a
concurrent.futures.ThreadPoolExecutor. We then set the max_workers
setting to the number of threads we want in our taskflow engine thread
pool.
2. During flow load, we define the engine to be 'parallel' (note:
'serial' is the default). This means that unordered flows will run in
parallel as opposed to serially.
3. As noted in the documentation[1], You can share an executor between
taskflow engines to share the thread pool.

Finally, you want to use "unordered" flows or sub-flows to execute
tasks concurrently.

[1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel

Michael

On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha at gmail.com> wrote:
>
> Hi,
>
> We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool.
>
> Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread.
>
> Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel.
>
> I came across iter_run method of the engine class. But not sure if that can be used for this purpose.
>
> Any help is highly appreciated.