[octavia][taskflow] Adaption of persistence/jobboard (taskflow expert review needed)
From what I see, taskflow expects that data, which will be stored in
Hello everyone! I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1]. I created a simple PoC to check what should be done to enable this functionality [2] . persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) Also functions that creates flow should be importable [4] (error [4.1]). These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions. As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5] I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something. Thank you for your time in advance! [1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backe... [3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers... [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy -- Regards, Ann Taraday Mirantis, Inc
Thank you Ann for working on this. It has been on our roadmap[1] for some time. Using Taskflow JobBoard would bring huge value to Octavia by allowing sub-flow resumption of tasks. I inquired about this in the oslo team meeting a few weeks ago and sadly it seems that most if not all of the taskflow experts are no longer working on OpenStack. This may mean "we" are the current Taskflow experts.... I also inquired about adding etcd as an option for the jobs engine implementation. Currently only Zookeeper and Redis are implemented. Etcd is attractive as it provides similar functionality (to my limited knowledge of what Taskflow needs) and is already an OpenStack base service[2]. This may be an additional chunk of work to make this a viable option. The refactor of the flow data storage from oslo.db/sqlalchemy data models aligns with some of the work we need to do to make the amphora driver a proper Octavia driver. Currently it doesn't fully use the provider driver interface data passing. This work could resolve two issues at the same time. It also looks like you have found a reasonable solution to the importable flows issue. I did include this on the topic list for the PTG[3] expecting we would need to discuss it there. I think we have a number of questions to answer on this topic. 1. Do we have resources to work on this? 2. Is Taskflow JobBoard the right solution? Is there alternative we could implement without the overhead of JobBoard? Maybe a hybrid approach is the right answer. 3. Are we ok with requiring either Zookeeper or Redis for this functionality? Do we need to implement a TaskFlow driver for etcd? 4. Should this be implemented as an alternate controller driver to the current implementation? (yes, even the controller is a driver in Octavia.) Are you planning to attend the PTG? If so we can work through these questions there, it is already on the agenda. If not, we should figure out either how to include you in that discussion, or continue the discussion on the mailing list. Michael [1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://governance.openstack.org/tc/reference/base-services.html [3] https://etherpad.openstack.org/p/octavia-train-ptg On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova@mirantis.com> wrote:
Hello everyone!
I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1]. I created a simple PoC to check what should be done to enable this functionality [2] .
From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) Also functions that creates flow should be importable [4] (error [4.1]).
These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions.
As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5]
I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something.
Thank you for your time in advance!
[1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backe... [3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers... [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy
-- Regards, Ann Taraday Mirantis, Inc
Thanks for your feedback! The good thing about implementation of taskflow approach is that backend type is set in configs and does not affect code. We can create config settings in a flexible way, so that operators could choose which backend is preferable for their cloud. Just have one option as default for devstack, testing, etc. Having etcd seems to be a good option, I did some experiments with it several years ago. But my concern here, if we do not have taskflow experts it may take a lot of time to implement it properly in taskflow. It is good to hear that some of refactor could be align with other activities and won't just disrupt the main course of work. Implementing all of this as an alternative controller driver is a great idea! In this case we can have it as experimental feature to gather some user feedback. Unfortunately, I'm not attending PTG, but I hope we will find a find to discuss this in IRC. On Wed, Apr 24, 2019 at 5:24 AM Michael Johnson <johnsomor@gmail.com> wrote:
Thank you Ann for working on this. It has been on our roadmap[1] for some time.
Using Taskflow JobBoard would bring huge value to Octavia by allowing sub-flow resumption of tasks.
I inquired about this in the oslo team meeting a few weeks ago and sadly it seems that most if not all of the taskflow experts are no longer working on OpenStack. This may mean "we" are the current Taskflow experts....
I also inquired about adding etcd as an option for the jobs engine implementation. Currently only Zookeeper and Redis are implemented. Etcd is attractive as it provides similar functionality (to my limited knowledge of what Taskflow needs) and is already an OpenStack base service[2]. This may be an additional chunk of work to make this a viable option.
The refactor of the flow data storage from oslo.db/sqlalchemy data models aligns with some of the work we need to do to make the amphora driver a proper Octavia driver. Currently it doesn't fully use the provider driver interface data passing. This work could resolve two issues at the same time.
It also looks like you have found a reasonable solution to the importable flows issue.
I did include this on the topic list for the PTG[3] expecting we would need to discuss it there. I think we have a number of questions to answer on this topic.
1. Do we have resources to work on this? 2. Is Taskflow JobBoard the right solution? Is there alternative we could implement without the overhead of JobBoard? Maybe a hybrid approach is the right answer. 3. Are we ok with requiring either Zookeeper or Redis for this functionality? Do we need to implement a TaskFlow driver for etcd? 4. Should this be implemented as an alternate controller driver to the current implementation? (yes, even the controller is a driver in Octavia.)
Are you planning to attend the PTG? If so we can work through these questions there, it is already on the agenda. If not, we should figure out either how to include you in that discussion, or continue the discussion on the mailing list.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://governance.openstack.org/tc/reference/base-services.html [3] https://etherpad.openstack.org/p/octavia-train-ptg
On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova@mirantis.com> wrote:
Hello everyone!
I was looking at the topic of usage taskflow persistence and jobboard in
I created a simple PoC to check what should be done to enable this functionality [2] .
From what I see, taskflow expects that data, which will be stored in
Also functions that creates flow should be importable [4] (error [4.1]).
These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions.
As far as I see the only OpenStack project which adapted taskflow
Octavia [1]. persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) persistence is poppy [5]
I'm looking for taskflow expect to take a look at all this and give some
comments - whether I am correct or missing something.
Thank you for your time in advance!
[1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] -
https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backe...
[3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers... [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy
-- Regards, Ann Taraday Mirantis, Inc
-- Regards, Ann Taraday Mirantis, Inc
Hi Octavia team, Thank you for the great discussion we had at the PTG and via video conference. I am super excited that we can start work on flow resumption. I have updated the Stroyboard story for the Jobboard work based on the discussion: https://storyboard.openstack.org/#!/story/2005072 I tried to break it down into parts that multiple people could work on in parallel. Please feel free to sign up for work you are interested in or to add additional tasks you might think of. Michael On Wed, Apr 24, 2019 at 6:15 AM Anna Taraday <akamyshnikova@mirantis.com> wrote:
Thanks for your feedback!
The good thing about implementation of taskflow approach is that backend type is set in configs and does not affect code. We can create config settings in a flexible way, so that operators could choose which backend is preferable for their cloud. Just have one option as default for devstack, testing, etc. Having etcd seems to be a good option, I did some experiments with it several years ago. But my concern here, if we do not have taskflow experts it may take a lot of time to implement it properly in taskflow.
It is good to hear that some of refactor could be align with other activities and won't just disrupt the main course of work. Implementing all of this as an alternative controller driver is a great idea! In this case we can have it as experimental feature to gather some user feedback.
Unfortunately, I'm not attending PTG, but I hope we will find a find to discuss this in IRC.
On Wed, Apr 24, 2019 at 5:24 AM Michael Johnson <johnsomor@gmail.com> wrote:
Thank you Ann for working on this. It has been on our roadmap[1] for some time.
Using Taskflow JobBoard would bring huge value to Octavia by allowing sub-flow resumption of tasks.
I inquired about this in the oslo team meeting a few weeks ago and sadly it seems that most if not all of the taskflow experts are no longer working on OpenStack. This may mean "we" are the current Taskflow experts....
I also inquired about adding etcd as an option for the jobs engine implementation. Currently only Zookeeper and Redis are implemented. Etcd is attractive as it provides similar functionality (to my limited knowledge of what Taskflow needs) and is already an OpenStack base service[2]. This may be an additional chunk of work to make this a viable option.
The refactor of the flow data storage from oslo.db/sqlalchemy data models aligns with some of the work we need to do to make the amphora driver a proper Octavia driver. Currently it doesn't fully use the provider driver interface data passing. This work could resolve two issues at the same time.
It also looks like you have found a reasonable solution to the importable flows issue.
I did include this on the topic list for the PTG[3] expecting we would need to discuss it there. I think we have a number of questions to answer on this topic.
1. Do we have resources to work on this? 2. Is Taskflow JobBoard the right solution? Is there alternative we could implement without the overhead of JobBoard? Maybe a hybrid approach is the right answer. 3. Are we ok with requiring either Zookeeper or Redis for this functionality? Do we need to implement a TaskFlow driver for etcd? 4. Should this be implemented as an alternate controller driver to the current implementation? (yes, even the controller is a driver in Octavia.)
Are you planning to attend the PTG? If so we can work through these questions there, it is already on the agenda. If not, we should figure out either how to include you in that discussion, or continue the discussion on the mailing list.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://governance.openstack.org/tc/reference/base-services.html [3] https://etherpad.openstack.org/p/octavia-train-ptg
On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova@mirantis.com> wrote:
Hello everyone!
I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1]. I created a simple PoC to check what should be done to enable this functionality [2] .
From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) Also functions that creates flow should be importable [4] (error [4.1]).
These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions.
As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5]
I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something.
Thank you for your time in advance!
[1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backe... [3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers... [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy
-- Regards, Ann Taraday Mirantis, Inc
-- Regards, Ann Taraday Mirantis, Inc
participants (2)
-
Anna Taraday
-
Michael Johnson