Open Stack

Thanks for your feedback!

The good thing about implementation of taskflow approach is that backend
type is set in configs and does not affect code. We can create config
settings in a flexible way, so that operators could choose which backend is
preferable for their cloud. Just have one option as default for devstack,
testing, etc.
Having etcd seems to be a good option, I did some experiments with it
several years ago. But my concern here, if we do not have taskflow experts
it may take a lot of time to implement it properly in taskflow.

It is good to hear that some of refactor could be align with other
activities and won't just disrupt the main course of work.
Implementing all of this as an alternative controller driver is a great
idea! In this case we can have it as experimental feature to gather some
user feedback.

Unfortunately, I'm not attending PTG, but I hope we will find a find to
discuss this in IRC.

On Wed, Apr 24, 2019 at 5:24 AM Michael Johnson <johnsomor at gmail.com> wrote:

> Thank you Ann for working on this. It has been on our roadmap[1] for some
> time.
>
> Using Taskflow JobBoard would bring huge value to Octavia by allowing
> sub-flow resumption of tasks.
>
> I inquired about this in the oslo team meeting a few weeks ago and
> sadly it seems that most if not all of the taskflow experts are no
> longer working on OpenStack. This may mean "we" are the current
> Taskflow experts....
>
> I also inquired about adding etcd as an option for the jobs engine
> implementation. Currently only Zookeeper and Redis are implemented.
> Etcd is attractive as it provides similar functionality (to my limited
> knowledge of what Taskflow needs) and is already an OpenStack base
> service[2]. This may be an additional chunk of work to make this a
> viable option.
>
> The refactor of the flow data storage from oslo.db/sqlalchemy data
> models aligns with some of the work we need to do to make the amphora
> driver a proper Octavia driver. Currently it doesn't fully use the
> provider driver interface data passing. This work could resolve two
> issues at the same time.
>
> It also looks like you have found a reasonable solution to the
> importable flows issue.
>
> I did include this on the topic list for the PTG[3] expecting we would
> need to discuss it there. I think we have a number of questions to
> answer on this topic.
>
> 1. Do we have resources to work on this?
> 2. Is Taskflow JobBoard the right solution? Is there alternative we
> could implement without the overhead of JobBoard? Maybe a hybrid
> approach is the right answer.
> 3. Are we ok with requiring either Zookeeper or Redis for this
> functionality? Do we need to implement a TaskFlow driver for etcd?
> 4. Should this be implemented as an alternate controller driver to the
> current implementation? (yes, even the controller is a driver in
> Octavia.)
>
> Are you planning to attend the PTG? If so we can work through these
> questions there, it is already on the agenda.
> If not, we should figure out either how to include you in that
> discussion, or continue the discussion on the mailing list.
>
> Michael
>
> [1] https://wiki.openstack.org/wiki/Octavia/Roadmap
> [2] https://governance.openstack.org/tc/reference/base-services.html
> [3] https://etherpad.openstack.org/p/octavia-train-ptg
>
> On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova at mirantis.com>
> wrote:
> >
> > Hello everyone!
> >
> > I was looking at the topic of usage taskflow persistence and jobboard in
> Octavia [1].
> > I created a simple PoC to check what should be done to enable this
> functionality [2] .
> >
> > From what I see, taskflow expects that data, which will be stored in
> persitence backend/jobboard backend, is a dict or an object easily
> converted to dicts [3] (error [3.1])
> > Also functions that creates flow should be importable [4] (error [4.1]).
> >
> > These two points lead to refactor required for Octavia to enable
> taskflow persistence and jobboard:
> > 1) Convert data which is passed between flows in dicts, at this moment
> it is db objects with links to other db objects.
> > 2) Create importable flow functions.
> >
> > As far as I see the only OpenStack project which adapted taskflow
> persistence is poppy [5]
> >
> > I'm looking for taskflow expect to take a look at all this and give some
> comments - whether I am correct or missing something.
> >
> > Thank you for your time in advance!
> >
> > [1] - https://storyboard.openstack.org/#!/story/2005072
> > [2] - https://review.openstack.org/#/c/647406
> > [3] -
> https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backends/impl_sqlalchemy.py#L458
> > [3.1] - http://paste.openstack.org/show/749530/
> > [4] -
> https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers.html#save_factory_details
> > [4.1] - http://paste.openstack.org/show/749527/
> > [5] - https://github.com/openstack/poppy
> >
> >
> > --
> > Regards,
> > Ann Taraday
> > Mirantis, Inc
>


-- 
Regards,
Ann Taraday
Mirantis, Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190424/ae8ccebc/attachment.html>