Open Stack

On Wed, 2015-04-22 at 12:36 +0000, Chen, Ken wrote:
> o	more complex workflows (job dependencies, DAGs, etc. Do we rely on Oozie, or something else?
>     >> Huichun is now figuring this. I am not whether you guys already have some detail ideas about this? If needed we can contribute some effort. If no details are ready, we can help draw a draft version first.

I just made a note on the pad 

https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions

Maybe the right approach here is to develop a mapping notation that can
be expressed as a JSON object (like the proposed job interface mapping).

If we can develop an abstract way to describe relationships between
jobs, then the individual EDP engines can implement it. For the Oozie
EDP engine, maybe it uses Oozie features in workflows.  For Spark, or
Storm, maybe it uses some existing opensource coordinator or one is
written.

The key idea would be to make job coordination part of the EDP engine,
with a well defined set of objects to describe the relationships.

What do you think? Just a rough idea.  Maybe there is a better way.