Open Stack

On 19 Mar 2014, at 02:08, Joshua Harlow <harlowja at yahoo-inc.com> wrote:

> is mistral planning doing this, throwing way the POC and rewriting it as a non-POC using the ideas learned from the POC?)[http://tinyurl.com/lbz293s]. 

Yes, if needed. I don’t think it’s too important and I agree that terminology here may be a thing that makes some confusion. If we decide most of the parts are fine to be just evolving (instead of rewriting them) we can use them as their are.

> For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm still not sure why u would want to make is_sync/is_async a primitive concept in a workflow system, shouldn't this be only up to the entity running the workflow to decide?


> Why is a task allowed to be sync/async

I would ask the opposite: “Why a task should always be sync?” Where does this limitation come from?

Any long lasting and/or resource intensive task should be logically considered async. Async here may not be a good word even. I apologize if it’s a problem for understanding. The main thing here is that a workflow system should be able not to hold local state to keep track of these heavy tasks. Otherwise, the consequences are well-known: hard to scale, hard to make it durable, etc etc. Some of the tasks may last weeks or even months (real life examples from the customers). If the engine is used as a hosted service then resource starvation may easily become an absolutely problem. From my experience, assumptions like that made at design phase used to make production systems go down. In other words, all the flowering variety of side-effects of Distributed Programming come from that.

So we decided to put this conceptual understanding into a core of the system.

Joshua, can we start working on that in TaskFlow? We’re ready to help and contribute (myself and other folks from the team).

> , that has major side-effects for state-persistence, resumption (and to me is a incorrect abstraction to provide) and general workflow execution control, I'd be very careful with this (which is why I am hesitant to add it without much much more discussion).

Of course, it has major effects. It’s what we’ve been emphasising as a fundamental difference. In that regard, one of the interesting ideas is to thing about the role that TaskFlow in OpenStack community. Should it be dealing with, say, with persistence not being a service and not being able to fully address HA? I know your point of view on that (that at least it could provide facilities to address HA partially etc), but that’s a topic to think about. I’d suggest we start discussing that too.

>> So we actually talked to people a lot (including Josh) and provided this reasoning when this question raised again. Reaction was nearly always positive and made a lot of sense to customers and developers.
>> 
>> Thought #2: A library shouldn't drive a project where it’s used.
> 
> 
> To me this assumes said library is fixed in stone, can't be changed, and can't be evolved. If a library is 'dead/vaporware' then sure, I would 100% agree with this, but all libraries in openstack do not fit into the later category; and those libraries can be evolved/developed/improved. As a community I think it is our goal to grow the libraries in the community (not reduce them or avoid them, as this is not benefical for the community). I think doug put this very well his essay @ http://tinyurl.com/lr9wvfl and imho we need to embrace evolving libraries in all projects, not avoiding them due to thoughts like this. 

100% agree that OS libraries are not fixed in stone, no questions. I meant a totally different thing: the process of evolution. TaskFlow is a beautiful well-written library, at least I think so. But we shouldn’t build a project around a library in the first place. We should be building a project and generating requirements to a library. If the library is ready to change it's wonderful. If it is not, I’m totally against of adjusting the project feature set so that it can be implemented using the library capabilities. Am I missing something?
For example, when I had a project where we used a persistence framework and it was ok for a while but at some point we realized that we needed lazy-loading that was missing in that framework and since the team of that framework said “it’s against our principles, you better change your vision of the project” we said ok, threw it away and started using Hibernate and eventually all became happy including the customer. As simple as that. And it makes a lot of sense to me. “Use whatever works well for you”.
At the same time, I clearly understand that in OpenStack we can’t do like that. And honestly, we don’t want, since eventually we’re working on the same huge thing: cloud. Well, actually we can ignore others but not for a long time :) Rule is a rule, TC folks honestly say it was done intentionally to make people collaborate, otherwise we would have hundreds of thousands similar projects and it would be much much harder to keep them and the entire OpenStack direction under control.

Joshua, honestly I thought we agreed that we needed to alter TaskFlow execution model to fit better into Mistral. That’s why we I have to refer to all these thoughts like “Library shouldn’t drive a project..” and so on. Just to make our point of view transparent and explain the real reason behind what you call “resistance in Mistral to change”. Specifically, I’m talking mostly again about active vs passive model (sync vs async). We can contribute, we can take our experience in Mistral and bring it to TaskFlow. In Mistral it’s already working though missing some certain minor things. There’s also a list of other things (conditionals, flexible persistence etc.) which we’re ready to talk about too but it doesn’t make a lot of sense to do right now.

>> Project needs should define the requirements to a library. Even being adopted in lots of places (this is apparently not true for TaskFlow right now which is fully understandable, a pretty young project as well) a library is just one of many tools used within a project. But not vice versa.
>> 
>> Thought #3. We 100% admit we’re not ready for incubation right now.
> 
> 
> This to me goes back to the POC question. I'm a pretty big believer that POC should be redone with the lessons learnt (and then whatever this is should be proposed for incubation when the team responsible for it believes it is ready for incubation).

Not sure I’m following here… Like a words game to me. “redone” doesn’t necessarily mean you should throw away all the codebase and you should use a different repo, different name, different team, different everything :)

> Sorry didn't mean to bomb more than usual ;)

It was more of a joke :) Np. Always glad to communicate with you.

Renat Akhmerov
@ Mirantis Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140319/ac5768e7/attachment.html>