Open Stack

Thank you Ann for working on this. It has been on our roadmap[1] for some time.

Using Taskflow JobBoard would bring huge value to Octavia by allowing
sub-flow resumption of tasks.

I inquired about this in the oslo team meeting a few weeks ago and
sadly it seems that most if not all of the taskflow experts are no
longer working on OpenStack. This may mean "we" are the current
Taskflow experts....

I also inquired about adding etcd as an option for the jobs engine
implementation. Currently only Zookeeper and Redis are implemented.
Etcd is attractive as it provides similar functionality (to my limited
knowledge of what Taskflow needs) and is already an OpenStack base
service[2]. This may be an additional chunk of work to make this a
viable option.

The refactor of the flow data storage from oslo.db/sqlalchemy data
models aligns with some of the work we need to do to make the amphora
driver a proper Octavia driver. Currently it doesn't fully use the
provider driver interface data passing. This work could resolve two
issues at the same time.

It also looks like you have found a reasonable solution to the
importable flows issue.

I did include this on the topic list for the PTG[3] expecting we would
need to discuss it there. I think we have a number of questions to
answer on this topic.

1. Do we have resources to work on this?
2. Is Taskflow JobBoard the right solution? Is there alternative we
could implement without the overhead of JobBoard? Maybe a hybrid
approach is the right answer.
3. Are we ok with requiring either Zookeeper or Redis for this
functionality? Do we need to implement a TaskFlow driver for etcd?
4. Should this be implemented as an alternate controller driver to the
current implementation? (yes, even the controller is a driver in
Octavia.)

Are you planning to attend the PTG? If so we can work through these
questions there, it is already on the agenda.
If not, we should figure out either how to include you in that
discussion, or continue the discussion on the mailing list.

Michael

[1] https://wiki.openstack.org/wiki/Octavia/Roadmap
[2] https://governance.openstack.org/tc/reference/base-services.html
[3] https://etherpad.openstack.org/p/octavia-train-ptg

On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova at mirantis.com> wrote:
>
> Hello everyone!
>
> I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1].
> I created a simple PoC to check what should be done to enable this functionality [2] .
>
> From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1])
> Also functions that creates flow should be importable [4] (error [4.1]).
>
> These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard:
> 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects.
> 2) Create importable flow functions.
>
> As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5]
>
> I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something.
>
> Thank you for your time in advance!
>
> [1] - https://storyboard.openstack.org/#!/story/2005072
> [2] - https://review.openstack.org/#/c/647406
> [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backends/impl_sqlalchemy.py#L458
> [3.1] - http://paste.openstack.org/show/749530/
> [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers.html#save_factory_details
> [4.1] - http://paste.openstack.org/show/749527/
> [5] - https://github.com/openstack/poppy
>
>
> --
> Regards,
> Ann Taraday
> Mirantis, Inc