<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@PMingLiU";
panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Andrew.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks for response. My reply in line.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Andrew Lazarev [mailto:alazarev@mirantis.com]
<br>
<b>Sent:</b> Saturday, March 21, 2015 12:10 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] About Sahara EDP New Ideas for Liberty<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Weiting,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">>1. Add a schedule feature to run the jobs on time:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">>This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> is a scheduler to help arrange the regular job to run.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">Looks like a great feature. And should be quite easy to implement. Feel free to create spec for that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">[Weiting] We are working on the spec and the bp has already been registered in
<a href="https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs">
https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs</a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal">>2. A more complex workflow design in Sahara EDP:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">>Current EDP only provide one job that is running on one cluster.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">Yes. And ability to run several jobs in one oozie workflow is discussed on every summit (e.g. 'coordinated jobs' at
<a href="https://etherpad.openstack.org/p/kilo-summit-sahara-edp">https://etherpad.openstack.org/p/kilo-summit-sahara-edp</a>). But for now it was not a priority<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">>But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it..<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">It means that workflow manager should be on Sahara side. Looks like a complicated feature. But we would be happy to help with designing and implementing it. Please file proposal for design session on ongoing summit. Are you going to Vancouver?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">[Weiting] I’m not sure I will be there because the plan is still not ready yet. We are also looking for some customer’s real case in big data area and see how they are using data processing in current environment.
However, for any idea we can update later. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">>Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If workflow is on Sahara side it should work automatically for all engines.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">[Weiting] Yes, agree.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Andrew.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Sun, Mar 8, 2015 at 3:17 AM, Chen, Weiting <<a href="mailto:weiting.chen@intel.com" target="_blank">weiting.chen@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi all.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We got several feedbacks about Sahara EDP’s future from some China customers.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Here are some ideas we would like to share with you and need your input if we can implement them in Sahara(Liberty).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1. Add a schedule feature to run the jobs on time:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2. A more complex workflow design in Sahara EDP:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Current EDP only provide one job that is running on one cluster.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">For example: Raw Data -> Job A(Cluster A) -> Job B(Cluster B) -> Job C(Cluster A) -> Result<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Actually in my opinion, this kind of job could be easy to implement by using Oozie as a workflow engine. But for current EDP, it doesn’t implement this kind of complex case.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">However, any suggestion is welcome.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>