Open Stack

Hi Octavia team,

Thank you for the great discussion we had at the PTG and via video
conference. I am super excited that we can start work on flow
resumption.

I have updated the Stroyboard story for the Jobboard work based on the
discussion: https://storyboard.openstack.org/#!/story/2005072

I tried to break it down into parts that multiple people could work on
in parallel. Please feel free to sign up for work you are interested
in or to add additional tasks you might think of.

Michael

On Wed, Apr 24, 2019 at 6:15 AM Anna Taraday <akamyshnikova at mirantis.com> wrote:
>
> Thanks for your feedback!
>
> The good thing about implementation of taskflow approach is that backend type is set in configs and does not affect code. We can create config settings in a flexible way, so that operators could choose which backend is preferable for their cloud. Just have one option as default for devstack, testing, etc.
> Having etcd seems to be a good option, I did some experiments with it several years ago. But my concern here, if we do not have taskflow experts it may take a lot of time to implement it properly in taskflow.
>
> It is good to hear that some of refactor could be align with other activities and won't just disrupt the main course of work.
> Implementing all of this as an alternative controller driver is a great idea! In this case we can have it as experimental feature to gather some user feedback.
>
> Unfortunately, I'm not attending PTG, but I hope we will find a find to discuss this in IRC.
>
> On Wed, Apr 24, 2019 at 5:24 AM Michael Johnson <johnsomor at gmail.com> wrote:
>>
>> Thank you Ann for working on this. It has been on our roadmap[1] for some time.
>>
>> Using Taskflow JobBoard would bring huge value to Octavia by allowing
>> sub-flow resumption of tasks.
>>
>> I inquired about this in the oslo team meeting a few weeks ago and
>> sadly it seems that most if not all of the taskflow experts are no
>> longer working on OpenStack. This may mean "we" are the current
>> Taskflow experts....
>>
>> I also inquired about adding etcd as an option for the jobs engine
>> implementation. Currently only Zookeeper and Redis are implemented.
>> Etcd is attractive as it provides similar functionality (to my limited
>> knowledge of what Taskflow needs) and is already an OpenStack base
>> service[2]. This may be an additional chunk of work to make this a
>> viable option.
>>
>> The refactor of the flow data storage from oslo.db/sqlalchemy data
>> models aligns with some of the work we need to do to make the amphora
>> driver a proper Octavia driver. Currently it doesn't fully use the
>> provider driver interface data passing. This work could resolve two
>> issues at the same time.
>>
>> It also looks like you have found a reasonable solution to the
>> importable flows issue.
>>
>> I did include this on the topic list for the PTG[3] expecting we would
>> need to discuss it there. I think we have a number of questions to
>> answer on this topic.
>>
>> 1. Do we have resources to work on this?
>> 2. Is Taskflow JobBoard the right solution? Is there alternative we
>> could implement without the overhead of JobBoard? Maybe a hybrid
>> approach is the right answer.
>> 3. Are we ok with requiring either Zookeeper or Redis for this
>> functionality? Do we need to implement a TaskFlow driver for etcd?
>> 4. Should this be implemented as an alternate controller driver to the
>> current implementation? (yes, even the controller is a driver in
>> Octavia.)
>>
>> Are you planning to attend the PTG? If so we can work through these
>> questions there, it is already on the agenda.
>> If not, we should figure out either how to include you in that
>> discussion, or continue the discussion on the mailing list.
>>
>> Michael
>>
>> [1] https://wiki.openstack.org/wiki/Octavia/Roadmap
>> [2] https://governance.openstack.org/tc/reference/base-services.html
>> [3] https://etherpad.openstack.org/p/octavia-train-ptg
>>
>> On Fri, Apr 19, 2019 at 6:16 AM Anna Taraday <akamyshnikova at mirantis.com> wrote:
>> >
>> > Hello everyone!
>> >
>> > I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1].
>> > I created a simple PoC to check what should be done to enable this functionality [2] .
>> >
>> > From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1])
>> > Also functions that creates flow should be importable [4] (error [4.1]).
>> >
>> > These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard:
>> > 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects.
>> > 2) Create importable flow functions.
>> >
>> > As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5]
>> >
>> > I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something.
>> >
>> > Thank you for your time in advance!
>> >
>> > [1] - https://storyboard.openstack.org/#!/story/2005072
>> > [2] - https://review.openstack.org/#/c/647406
>> > [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/backends/impl_sqlalchemy.py#L458
>> > [3.1] - http://paste.openstack.org/show/749530/
>> > [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helpers.html#save_factory_details
>> > [4.1] - http://paste.openstack.org/show/749527/
>> > [5] - https://github.com/openstack/poppy
>> >
>> >
>> > --
>> > Regards,
>> > Ann Taraday
>> > Mirantis, Inc
>
>
>
> --
> Regards,
> Ann Taraday
> Mirantis, Inc