<div dir="ltr">I like idea of "<span style="font-family:arial,sans-serif;font-size:13px">edp." prefix.</span><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Andrew.</span></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 29, 2014 at 6:23 AM, Trevor McKay <span dir="ltr"><<a href="mailto:tmckay@redhat.com" target="_blank">tmckay@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">So, assuming we go forward with this, the followup question is whether<br>
or not to move "main_class" and "java_opts" for Java actions into<br>
"edp.java.main_class" and "edp.java.java_opts" configs.<br>
<br>
I think yes.<br>
<br>
Best,<br>
<br>
Trevor<br>
<br>
On Wed, 2014-01-29 at 09:15 -0500, Trevor McKay wrote:<br>
> On Wed, 2014-01-29 at 14:35 +0400, Alexander Ignatov wrote:<br>
> > Thank you for bringing this up, Trevor.<br>
> ><br>
> > EDP gets more diverse and it's time to change its model.<br>
> > I totally agree with your proposal, but one minor comment.<br>
> > Instead of "savanna." prefix in job_configs wouldn't it be better to make it<br>
> > as "edp."? I think "savanna." is too more wide word for this.<br>
><br>
> +1, brilliant. EDP is perfect. I was worried about the scope of<br>
> "savanna." too.<br>
><br>
> > And one more bureaucratic thing... I see you already started implementing it [1],<br>
> > and it is named and goes as new EDP workflow [2]. I think new bluprint should be<br>
> > created for this feature to track all code changes as well as docs updates.<br>
> > Docs I mean public Savanna docs about EDP, rest api docs and samples.<br>
><br>
> Absolutely, I can make it new blueprint. Thanks.<br>
><br>
> > [1] <a href="https://review.openstack.org/#/c/69712" target="_blank">https://review.openstack.org/#/c/69712</a><br>
> > [2] <a href="https://blueprints.launchpad.net/openstack/?searchtext=edp-oozie-streaming-mapreduce" target="_blank">https://blueprints.launchpad.net/openstack/?searchtext=edp-oozie-streaming-mapreduce</a><br>
> ><br>
> > Regards,<br>
> > Alexander Ignatov<br>
> ><br>
> ><br>
> ><br>
> > On 28 Jan 2014, at 20:47, Trevor McKay <<a href="mailto:tmckay@redhat.com">tmckay@redhat.com</a>> wrote:<br>
> ><br>
> > > Hello all,<br>
> > ><br>
> > > In our first pass at EDP, the model for job settings was very consistent<br>
> > > across all of our job types. The execution-time settings fit into this<br>
> > > (superset) structure:<br>
> > ><br>
> > > job_configs = {'configs': {}, # config settings for oozie and hadoop<br>
> > > 'params': {}, # substitution values for Pig/Hive<br>
> > > 'args': []} # script args (Pig and Java actions)<br>
> > ><br>
> > > But we have some things that don't fit (and probably more in the<br>
> > > future):<br>
> > ><br>
> > > 1) Java jobs have 'main_class' and 'java_opts' settings<br>
> > > Currently these are handled as additional fields added to the<br>
> > > structure above. These were the first to diverge.<br>
> > ><br>
> > > 2) Streaming MapReduce (anticipated) requires mapper and reducer<br>
> > > settings (different than the mapred.xxxx.class settings for<br>
> > > non-streaming MapReduce)<br>
> > ><br>
> > > Problems caused by adding fields<br>
> > > --------------------------------<br>
> > > The job_configs structure above is stored in the database. Each time we<br>
> > > add a field to the structure above at the level of configs, params, and<br>
> > > args, we force a change to the database tables, a migration script and a<br>
> > > change to the JSON validation for the REST api.<br>
> > ><br>
> > > We also cause a change for python-savannaclient and potentially other<br>
> > > clients.<br>
> > ><br>
> > > This kind of change seems bad.<br>
> > ><br>
> > > Proposal: Borrow a page from Oozie and add "savanna." configs<br>
> > > -------------------------------------------------------------<br>
> > > I would like to fit divergent job settings into the structure we already<br>
> > > have. One way to do this is to leverage the 'configs' dictionary. This<br>
> > > dictionary primarily contains settings for hadoop, but there are a<br>
> > > number of "oozie.xxx" settings that are passed to oozie as configs or<br>
> > > set by oozie for the benefit of running apps.<br>
> > ><br>
> > > What if we allow "savanna." settings to be added to configs? If we do<br>
> > > that, any and all special configuration settings for specific job types<br>
> > > or subtypes can be handled with no database changes and no api changes.<br>
> > ><br>
> > > Downside<br>
> > > --------<br>
> > > Currently, all 'configs' are rendered in the generated oozie workflow.<br>
> > > The "savanna." settings would be stripped out and processed by Savanna,<br>
> > > thereby changing that behavior a bit (maybe not a big deal)<br>
> > ><br>
> > > We would also be mixing "savanna." configs with config_hints for jobs,<br>
> > > so users would potentially see "savanna.xxxx" settings mixed with oozie<br>
> > > and hadoop settings. Again, maybe not a big deal, but it might blur the<br>
> > > lines a little bit. Personally, I'm okay with this.<br>
> > ><br>
> > > Slightly different<br>
> > > ------------------<br>
> > > We could also add a "'savanna-configs': {}" element to job_configs to<br>
> > > keep the configuration spaces separate.<br>
> > ><br>
> > > But, now we would have 'savanna-configs' (or another name), 'configs',<br>
> > > 'params', and 'args'. Really? Just how many different types of values<br>
> > > can we come up with? :)<br>
> > ><br>
> > > I lean away from this approach.<br>
> > ><br>
> > > Related: breaking up the superset<br>
> > > ---------------------------------<br>
> > ><br>
> > > It is also the case that not every job type has every value type.<br>
> > ><br>
> > > Configs Params Args<br>
> > > Hive Y Y N<br>
> > > Pig Y Y Y<br>
> > > MapReduce Y N N<br>
> > > Java Y N Y<br>
> > ><br>
> > > So do we make that explicit in the docs and enforce it in the api with<br>
> > > errors?<br>
> > ><br>
> > > Thoughts? I'm sure there are some :)<br>
> > ><br>
> > > Best,<br>
> > ><br>
> > > Trevor<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > OpenStack-dev mailing list<br>
> > > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>