Open Stack

Since the uploading of JS is an interesting concept, let me just share a little about another service at Y! that has done this.

http://developer.yahoo.com/yql/guide/yql-execute-chapter.html

YQL (although closed-source) has exposed this ability, so its entirely possible to do (although it likely won't be running in python). YQL itself is slightly similar to a DSL also (its DSL is similar to SQL). The nice thing about using something like JS (to continue along this idea, even if nobody wants to actually implement this) is that things like rhino<http://en.wikipedia.org/wiki/Rhino_%28JavaScript_engine%29> do provide execution limits (at the instruction level). Of course this would potentially bring in java (I am guessing node.js can do something similar as rhino). Anyways…

To further this lets continue working on https://etherpad.openstack.org/p/taskflow-mistral and see if we can align somehow (I hope it's not to late to do this, seeing that there appears to be a lot of resistance from the mistral community to change). But I agree with clint, and hope that we can have a healthy collaboration as a community instead of being in competing silos (which is not healthy).

-Josh

From: Clint Byrum <clint at fewbar.com<mailto:clint at fewbar.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Monday, March 17, 2014 at 9:41 AM
To: openstack-dev <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Mistral] Actions design BP

Excerpts from Renat Akhmerov's message of 2014-03-17 07:35:02 -0700:
On 16 Mar 2014, at 19:05, Clint Byrum <clint at fewbar.com<mailto:clint at fewbar.com>> wrote:
> From my perspective, as somebody who is considering Mistral for some
> important things, the fact that the taskflow people are not aligned with
> the Mistral people tells me that Mistral is not what I thought it was:
> taskflow for direct user consumption.
Yes, it was just an initial idea we were trying to pursue. As we moved forward we understood it was too limited and had a lot of pitfalls. The reason is the key difference between library and service. Making something a service flips everything upside down in terms of requirements to this new system. The logical questions/implications are:
If that’s a service why should its REST API be Python oriented? Bindings - yes (it gives a certain convenience) but that’s a different story..
A whole set of questions related to how we distribute Python-written tasks (especially in multi tenant environment, e.g. as a cloud hosted service):
Dependencies
Sandboxing
Result serialisation
etc.
It becomes logical to be able to use it with non-python clients and external systems.
Orientation to long-lived processes(workflows). Synchronous execution model no longer works well, instead de facto asynchronous event-based architecture fits better. I know it may sound too generic, it’s a whole different topic..
It becomes logical to provide more high-level capabilities rather than a library does (e.g. Horizon Dashboard. It would not be much of use if the service was based on Python).

I assume Mistral is written in Python though, and so it should be using
Taskflow for its own workflow. I understand though, that to run _user_
workflows you can't just expect them to upload python or always run
python as Mistral wouldn't do much for them at that point.

However, for the User<->Taskflow integration, I think Josh Harlow offered
a really reasonable suggestion to that, which is instead of inventing
a DSL (frankly, I'm pretty frustrated with the Heat DSL's we have, so I
may be biased here), just embrace javascript or lua, which are designed
to be embedded and executed in a less-than-trusted context.

I think it would be a great story for users if Mistral worked like this:

- Upload javascript expression of workflow, with external callouts for
  complicated things.
- Run code that uses Mistral API to poll-for or subscribe-to
  notifications waiting for instructions from Mistral when it is
  supposed to run said external callouts, and feeds back data.

I believe it’s not the full list.
Despite of all this I’m not against of using TaskFlow in implementation, at least some pieces of it. My only call is: it should be really beneficial rather than bringing pain. So I’m ok if we keep aligning our capabilities, roadmaps, terminology and whatever else is needed.
Btw, it would be really helpful if you could clarify what you meant “some important things”. Asking because I still feel like we could have much more input from potential users. Any information is appreciated.

- Heat needs to move from being a single state machine (heat-engine owns
  all of the running tasks for a stack in one engine at a time) to a
  distributed state machine. Before we do that, we need to consider how
  Heat expresses workflow. If Mistral were a distributed workflow
  engine, it would make a lot of sense for Heat to make use of it for
  this purpose.

- TripleO deploys a number of things that need distributed workflow.
  I think at this point the people involved with that are looking more
  at lower level tools like RAFT and Concoord. But once the distributed
  state machine is settled, there will be a need to express the
  distributed workflow. I'm disinclined to diverge from Taskflow, even
  though I am quite inclined to embrace API's.

> So, while it may be annoying to have your "day to day project
> activities" questioned, it is pretty core to the discussion considering
> that this suggests several things that diverge from taskflow's core
> model.
Np. I still keep finding things/rules in OpenStack that I’m not really aware of or didn’t get used to yet. If that’s not how it should be done in OS then it’s fine.

I'm sorry if the message was harsh, but I see this happening a lot.

I don't think it is a rule. I think the principle here is to collaborate
on things that should be aligned. If Taskflow doesn't do what you need
it to do, then I suggest _fixing that_ rather than writing a private
implementation. It will make life better for all users of taskflow and
it will keep Mistral extremely simple, which will help with adoption by
operators _and_ users.

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140317/4b959d90/attachment.html>