Open Stack

Excerpts from Renat Akhmerov's message of 2014-03-17 07:35:02 -0700:
> 
> On 16 Mar 2014, at 19:05, Clint Byrum <clint at fewbar.com> wrote:
> 
> > From my perspective, as somebody who is considering Mistral for some
> > important things, the fact that the taskflow people are not aligned with
> > the Mistral people tells me that Mistral is not what I thought it was:
> > taskflow for direct user consumption.
> 
> Yes, it was just an initial idea we were trying to pursue. As we moved forward we understood it was too limited and had a lot of pitfalls. The reason is the key difference between library and service. Making something a service flips everything upside down in terms of requirements to this new system. The logical questions/implications are:
> If that’s a service why should its REST API be Python oriented? Bindings - yes (it gives a certain convenience) but that’s a different story..
> A whole set of questions related to how we distribute Python-written tasks (especially in multi tenant environment, e.g. as a cloud hosted service):
> Dependencies
> Sandboxing
> Result serialisation
> etc.
> It becomes logical to be able to use it with non-python clients and external systems.
> Orientation to long-lived processes(workflows). Synchronous execution model no longer works well, instead de facto asynchronous event-based architecture fits better. I know it may sound too generic, it’s a whole different topic..
> It becomes logical to provide more high-level capabilities rather than a library does (e.g. Horizon Dashboard. It would not be much of use if the service was based on Python).
> 

I assume Mistral is written in Python though, and so it should be using
Taskflow for its own workflow. I understand though, that to run _user_
workflows you can't just expect them to upload python or always run
python as Mistral wouldn't do much for them at that point.

However, for the User<->Taskflow integration, I think Josh Harlow offered
a really reasonable suggestion to that, which is instead of inventing
a DSL (frankly, I'm pretty frustrated with the Heat DSL's we have, so I
may be biased here), just embrace javascript or lua, which are designed
to be embedded and executed in a less-than-trusted context.

I think it would be a great story for users if Mistral worked like this:

- Upload javascript expression of workflow, with external callouts for
  complicated things.
- Run code that uses Mistral API to poll-for or subscribe-to
  notifications waiting for instructions from Mistral when it is
  supposed to run said external callouts, and feeds back data.

> I believe it’s not the full list.
> 
> Despite of all this I’m not against of using TaskFlow in implementation, at least some pieces of it. My only call is: it should be really beneficial rather than bringing pain. So I’m ok if we keep aligning our capabilities, roadmaps, terminology and whatever else is needed. 
> 
> Btw, it would be really helpful if you could clarify what you meant “some important things”. Asking because I still feel like we could have much more input from potential users. Any information is appreciated.
> 

- Heat needs to move from being a single state machine (heat-engine owns
  all of the running tasks for a stack in one engine at a time) to a
  distributed state machine. Before we do that, we need to consider how
  Heat expresses workflow. If Mistral were a distributed workflow
  engine, it would make a lot of sense for Heat to make use of it for
  this purpose.

- TripleO deploys a number of things that need distributed workflow.
  I think at this point the people involved with that are looking more
  at lower level tools like RAFT and Concoord. But once the distributed
  state machine is settled, there will be a need to express the
  distributed workflow. I'm disinclined to diverge from Taskflow, even
  though I am quite inclined to embrace API's.

> > So, while it may be annoying to have your "day to day project
> > activities" questioned, it is pretty core to the discussion considering
> > that this suggests several things that diverge from taskflow's core
> > model.
> 
> Np. I still keep finding things/rules in OpenStack that I’m not really aware of or didn’t get used to yet. If that’s not how it should be done in OS then it’s fine.

I'm sorry if the message was harsh, but I see this happening a lot.

I don't think it is a rule. I think the principle here is to collaborate
on things that should be aligned. If Taskflow doesn't do what you need
it to do, then I suggest _fixing that_ rather than writing a private
implementation. It will make life better for all users of taskflow and
it will keep Mistral extremely simple, which will help with adoption by
operators _and_ users.