Open Stack

Hi Andrew.

Thanks for response. My reply in line.

From: Andrew Lazarev [mailto:alazarev at mirantis.com]
Sent: Saturday, March 21, 2015 12:10 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

Hi Weiting,

>1. Add a schedule feature to run the jobs on time:
>This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there
> is a scheduler to help arrange the regular job to run.
Looks like a great feature. And should be quite easy to implement. Feel free to create spec for that.
[Weiting] We are working on the spec and the bp has already been registered in https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs.

>2. A more complex workflow design in Sahara EDP:
>Current EDP only provide one job that is running on one cluster.
Yes. And ability to run several jobs in one oozie workflow is discussed on every summit (e.g. 'coordinated jobs' at https://etherpad.openstack.org/p/kilo-summit-sahara-edp). But for now it was not a priority

>But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it..
It means that workflow manager should be on Sahara side. Looks like a complicated feature. But we would be happy to help with designing and implementing it. Please file proposal for design session on ongoing summit. Are you going to Vancouver?
[Weiting] I’m not sure I will be there because the plan is still not ready yet. We are also looking for some customer’s real case in big data area and see how they are using data processing in current environment. However, for any idea we can update later.

>Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios.
If workflow is on Sahara side it should work automatically for all engines.
[Weiting] Yes, agree.

Thanks,
Andrew.



On Sun, Mar 8, 2015 at 3:17 AM, Chen, Weiting <weiting.chen at intel.com<mailto:weiting.chen at intel.com>> wrote:
Hi all.

We got several feedbacks about Sahara EDP’s future from some China customers.
Here are some ideas we would like to share with you and need your input if we can implement them in Sahara(Liberty).

1. Add a schedule feature to run the jobs on time:
This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run.

2. A more complex workflow design in Sahara EDP:
Current EDP only provide one job that is running on one cluster.
But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it.
For example: Raw Data -> Job A(Cluster A) -> Job B(Cluster B) -> Job C(Cluster A) -> Result
Actually in my opinion, this kind of job could be easy to implement by using Oozie as a workflow engine. But for current EDP, it doesn’t implement this kind of complex case.
Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios.

However, any suggestion is welcome.
Thanks.


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150324/6c3fa0bf/attachment.html>