Open Stack

From: Renat Akhmerov <rakhmerov at mirantis.com<mailto:rakhmerov at mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Monday, March 17, 2014 at 10:51 PM
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Mistral] Actions design BP

On 18 Mar 2014, at 01:32, Joshua Harlow <harlowja at yahoo-inc.com<mailto:harlowja at yahoo-inc.com>> wrote:
To further this lets continue working on https://etherpad.openstack.org/p/taskflow-mistral and see if we can align somehow

Sure.

(I hope it's not to late to do this,

Never late IMO.

seeing that there appears to be a lot of resistance from the mistral community to change.

Could you please let us know how you made this conclusion? I’m frankly surprised.. It’s not just my curiosity or something, I want to keep improving in what I’m doing.

So maybe its just my misinterpretation or miscommunication, but certain discussions like @ http://tinyurl.com/m395o2r<http://lists.openstack.org/pipermail/openstack-dev/2014-March/029983.html> (mention of 'inalienable parts of Mistral', 'Joshua propose detailed Mistral design based on TaskFlow') seem to be what causes me to think that what mistral has been making is more of a POC and is actually an implementation. To me that means the mistral project is already way past the POC mode (imho POC's are meant to explore concepts, then be thrown away and reimplemented as a real project, is mistral planning doing this, throwing way the POC and rewriting it as a non-POC using the ideas learned from the POC?)[http://tinyurl.com/lbz293s].



Generally, just to clarify the situation let me provide our vision of what we’re doing at the very high level.

As mentioned many times, we’re still building a PoC. Yes, it turned out to take longer which is totally fine since we’ve done a lot of research, lots of coding exercises, talks, discussions with our customers. We’ve involved several new contributors from two different companies, they have their requirements and use cases too. We’ve gathered a lot of specific requirements to what should be a workflow engine. This all was the exact intention of that phase of the project: understand better what we should build. If you look at Mistral list of blueprints you’ll see around 40 of them where 80-90% of them come from real needs of real projects. And not everything is still captured in BPs because something is still not shaped well enough in our minds.

Thought #1: In POC we’ve been concentrating on use cases and requirements. Implementation has been secondary.

Sure that’s fine that’s what a POC is for. See above.


TaskFlow or anything else just hasn’t mattered a lot so far. But, at the same time, I want to remind that in December we tried to use TaskFlow to implement the very basic functionality in Mistral (only dependency based model). Honestly, we failed to produce a result that we would be satisfied with since TaskFlow lacked, for example, the ability to run tasks in an asynchronous manner. This was not a problem at all, this is the real world. So I created a BP to address that problem in TaskFlow ([0]). So we decided to proceed with it with an intent to rejoin later.
And may be even the most important reason not to use TaskFlow was that we did want to have a clear research. We found that less productive to try to build a project around an existing library than concentrating on use cases and high-level requirements. From my experience, it never works well since in your thinking you always stick to limitations of that lib and assumptions made in it.

For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm still not sure why u would want to make is_sync/is_async a primitive concept in a workflow system, shouldn't this be only up to the entity running the workflow to decide? Why is a task allowed to be sync/async, that has major side-effects for state-persistence, resumption (and to me is a incorrect abstraction to provide) and general workflow execution control, I'd be very careful with this (which is why I am hesitant to add it without much much more discussion).


So we actually talked to people a lot (including Josh) and provided this reasoning when this question raised again. Reaction was nearly always positive and made a lot of sense to customers and developers.

Thought #2: A library shouldn't drive a project where it’s used.

To me this assumes said library is fixed in stone, can't be changed, and can't be evolved. If a library is 'dead/vaporware' then sure, I would 100% agree with this, but all libraries in openstack do not fit into the later category; and those libraries can be evolved/developed/improved. As a community I think it is our goal to grow the libraries in the community (not reduce them or avoid them, as this is not benefical for the community). I think doug put this very well his essay @ http://tinyurl.com/lr9wvfl and imho we need to embrace evolving libraries in all projects, not avoiding them due to thoughts like this.


Project needs should define the requirements to a library. Even being adopted in lots of places (this is apparently not true for TaskFlow right now which is fully understandable, a pretty young project as well) a library is just one of many tools used within a project. But not vice versa.

Thought #3. We 100% admit we’re not ready for incubation right now.

This to me goes back to the POC question. I'm a pretty big believer that POC should be redone with the lessons learnt (and then whatever this is should be proposed for incubation when the team responsible for it believes it is ready for incubation).


We prepared an incubation request already with most of the formal requirements met by the project. And even thought the interest to Mistral is serious (at least 5-6 projects intent to use it, one already started playing and prototyping with Mistral) we want to be honest with the community in that we’re not really ready for incubation since even our own solid understanding of core requirements and API/DSL is only on its way. This is just one more argument that spending significant time on thinking how to fit TaskFlow into our unstable Mistral vision is not affordable for us at the moment. Despite of that, last week we started spending that time after Josh started bombing us with his concerns :).

Sorry didn't mean to bomb more than usual ;)



If that’s all harsh, sorry. We’re interested in keeping in touch with Josh and others.

[0] https://blueprints.launchpad.net/taskflow/+spec/async-tasks

Renat Akhmerov
@ Mirantis Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140318/ff6e5e70/attachment.html>