[openstack-dev] About Sahara EDP New Ideas for Liberty
tmckay at redhat.com
Wed Apr 22 13:59:21 UTC 2015
On Wed, 2015-04-22 at 12:36 +0000, Chen, Ken wrote:
> o more complex workflows (job dependencies, DAGs, etc. Do we rely on Oozie, or something else?
> >> Huichun is now figuring this. I am not whether you guys already have some detail ideas about this? If needed we can contribute some effort. If no details are ready, we can help draw a draft version first.
I just made a note on the pad
Maybe the right approach here is to develop a mapping notation that can
be expressed as a JSON object (like the proposed job interface mapping).
If we can develop an abstract way to describe relationships between
jobs, then the individual EDP engines can implement it. For the Oozie
EDP engine, maybe it uses Oozie features in workflows. For Spark, or
Storm, maybe it uses some existing opensource coordinator or one is
The key idea would be to make job coordination part of the EDP engine,
with a well defined set of objects to describe the relationships.
What do you think? Just a rough idea. Maybe there is a better way.
More information about the OpenStack-dev