Open Stack

thanks Michael,

I am aware that executor can be reused by different engines.
My query was regarding if multiple engines can share same thread for
running the engines(and not the tasks of those engines).

I tried run_iter which can be used to run multiple engines but the tasks of
individual engines are run one after another.
Probably engine is holding on to the thread.

This is limiting our ability run multiple workflows (i.e. engines) in
parallel.

My query is - is it possible to run multiple engines in parallel on same
thread (using some asynchronous task execution)


On Thu, Feb 27, 2020 at 3:18 AM Michael Johnson <johnsomor at gmail.com> wrote:

> Hi Sachin,
>
> I'm not 100% sure I understand your need, but I will attempt to answer
> and you can correct me if I am off base.
>
> Taskflow engines (you can create as many of these as you want) use an
> executor defined at flow load time.
>
> Here is a snippet from the Octavia code:
>         self.executor = concurrent.futures.ThreadPoolExecutor(
>             max_workers=CONF.task_flow.max_workers)
>         eng = tf_engines.load(
>             flow,
>             engine=CONF.task_flow.engine,
>             executor=self.executor,
>             never_resolve=CONF.task_flow.disable_revert,
>             **kwargs)
>
> The parts you are likely interested in are:
>
> 1. The executor. In this case we are using a
> concurrent.futures.ThreadPoolExecutor. We then set the max_workers
> setting to the number of threads we want in our taskflow engine thread
> pool.
> 2. During flow load, we define the engine to be 'parallel' (note:
> 'serial' is the default). This means that unordered flows will run in
> parallel as opposed to serially.
> 3. As noted in the documentation[1], You can share an executor between
> taskflow engines to share the thread pool.
>
> Finally, you want to use "unordered" flows or sub-flows to execute
> tasks concurrently.
>
> [1] https://docs.openstack.org/taskflow/latest/user/engines.html#parallel
>
> Michael
>
> On Wed, Feb 26, 2020 at 7:19 AM Sachin Laddha <sachinladdha at gmail.com>
> wrote:
> >
> > Hi,
> >
> > We are using taskflow to execute workflows. Each workflow is executed by
> a separate thread (using engine.run() method). This is limiting our
> capability to execute maximum number of workflows that can run in parallel.
> It is limited by the number of threads there in the thread pool.
> >
> > Most of the time, the workflow tasks are run by agents which could take
> some time to complete. Each engine is alive and runs on a dedicated thread.
> >
> > Is there any way to reuse or run multiple engines on one thread. The
> individual tasks of these engines can run in parallel.
> >
> > I came across iter_run method of the engine class. But not sure if that
> can be used for this purpose.
> >
> > Any help is highly appreciated.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200227/ed83e4f8/attachment.html>