Open Stack

Hi all.

We got several feedbacks about Sahara EDP's future from some China customers.
Here are some ideas we would like to share with you and need your input if we can implement them in Sahara(Liberty).

1. Add a schedule feature to run the jobs on time:
This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run.

2. A more complex workflow design in Sahara EDP:
Current EDP only provide one job that is running on one cluster.
But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it.
For example: Raw Data -> Job A(Cluster A) -> Job B(Cluster B) -> Job C(Cluster A) -> Result
Actually in my opinion, this kind of job could be easy to implement by using Oozie as a workflow engine. But for current EDP, it doesn't implement this kind of complex case.
Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios.

However, any suggestion is welcome.
Thanks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150308/abc5fd30/attachment.html>