Open Stack

Hi Weiting,

>1. Add a schedule feature to run the jobs on time:
>This request comes from the customer, they usually run the job in a
specific time every day. So it should be great if there
> is a scheduler to help arrange the regular job to run.
Looks like a great feature. And should be quite easy to implement. Feel
free to create spec for that.

>2. A more complex workflow design in Sahara EDP:
>Current EDP only provide one job that is running on one cluster.
Yes. And ability to run several jobs in one oozie workflow is discussed on
every summit (e.g. 'coordinated jobs' at
https://etherpad.openstack.org/p/kilo-summit-sahara-edp). But for now it
was not a priority

>But in a real case, it should be more complex, they usually use multiple
jobs to calculate the data and may use several different type clusters to
process it..
It means that workflow manager should be on Sahara side. Looks like a
complicated feature. But we would be happy to help with designing and
implementing it. Please file proposal for design session on ongoing summit.
Are you going to Vancouver?

>Another concern is about Spark, for Spark it cannot use Oozie to do this.
So we need to create an abstract layer to help to implement this kind of
scenarios.
If workflow is on Sahara side it should work automatically for all engines.

Thanks,
Andrew.



On Sun, Mar 8, 2015 at 3:17 AM, Chen, Weiting <weiting.chen at intel.com>
wrote:

>  Hi all.
>
>
>
> We got several feedbacks about Sahara EDP’s future from some China
> customers.
>
> Here are some ideas we would like to share with you and need your input if
> we can implement them in Sahara(Liberty).
>
>
>
> 1. Add a schedule feature to run the jobs on time:
>
> This request comes from the customer, they usually run the job in a
> specific time every day. So it should be great if there is a scheduler to
> help arrange the regular job to run.
>
>
>
> 2. A more complex workflow design in Sahara EDP:
>
> Current EDP only provide one job that is running on one cluster.
>
> But in a real case, it should be more complex, they usually use multiple
> jobs to calculate the data and may use several different type clusters to
> process it.
>
> For example: Raw Data -> Job A(Cluster A) -> Job B(Cluster B) -> Job
> C(Cluster A) -> Result
>
> Actually in my opinion, this kind of job could be easy to implement by
> using Oozie as a workflow engine. But for current EDP, it doesn’t implement
> this kind of complex case.
>
> Another concern is about Spark, for Spark it cannot use Oozie to do this.
> So we need to create an abstract layer to help to implement this kind of
> scenarios.
>
>
>
> However, any suggestion is welcome.
>
> Thanks.
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150320/4fe43128/attachment.html>