Thank you Ann for working on this. It has been on our roadmap[1] for some time. Using Taskflow JobBoard would bring huge value to Octavia by allowing sub-flow resumption of tasks. I inquired about this in the oslo team meeting a few weeks ago and sadly it seems that most if not all of the taskflow experts are no longer working on OpenStack. This may mean "we" are the current Taskflow experts.... I also inquired about adding etcd as an option for the jobs engine implementation. Currently only Zookeeper and Redis are implemented. Etcd is attractive as it provides similar functionality (to my limited knowledge of what Taskflow needs) and is already an OpenStack base service[2]. This may be an additional chunk of work to make this a viable option. The refactor of the flow data storage from oslo.db/sqlalchemy data models aligns with some of the work we need to do to make the amphora driver a proper Octavia driver. Currently it doesn't fully use the provider driver interface data passing. This work could resolve two issues at the same time. It also looks like you have found a reasonable solution to the importable flows issue. I did include this on the topic list for the PTG[3] expecting we would need to discuss it there. I think we have a number of questions to answer on this topic. 1. Do we have resources to work on this? 2. Is Taskflow JobBoard the right solution? Is there alternative we could implement without the overhead of JobBoard? Maybe a hybrid approach is the right answer. 3. Are we ok with requiring either Zookeeper or Redis for this functionality? Do we need to implement a TaskFlow driver for etcd? 4. Should this be implemented as an alternate controller driver to the current implementation? (yes, even the controller is a driver in Octavia.) Are you planning to attend the PTG? If so we can work through these questions there, it is already on the agenda. If not, we should figure out either how to include you in that discussion, or continue the discussion on the mailing list. Michael [1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://governance.openstack.org/tc/reference/base-services.html [3] https://etherpad.openstack.org/p/octavia-train-ptg On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova@mirantis.com> wrote:
Hello everyone!
I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1]. I created a simple PoC to check what should be done to enable this functionality [2] .
From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) Also functions that creates flow should be importable [4] (error [4.1]).
These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions.
As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5]
I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something.
Thank you for your time in advance!
[1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backe... [3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers... [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy
-- Regards, Ann Taraday Mirantis, Inc