<div dir="ltr">Trevor,<div><br></div><div>it sounds reasonable to move main_class and java_opts to edp.java.</div><div><br></div><div>Jon,</div><div><br></div><div>does you mean neutron-related info for namespaces support? If yes than neutron isn't the user-side config.</div>
<div><br></div><div>Thanks.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 29, 2014 at 6:37 PM, Jon Maron <span dir="ltr"><<a href="mailto:jmaron@hortonworks.com" target="_blank">jmaron@hortonworks.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I imagine ‘neutron’ would follow suit as well..<br>
<div class="HOEnZb"><div class="h5"><br>
On Jan 29, 2014, at 9:23 AM, Trevor McKay <<a href="mailto:tmckay@redhat.com">tmckay@redhat.com</a>> wrote:<br>
<br>
> So, assuming we go forward with this, the followup question is whether<br>
> or not to move "main_class" and "java_opts" for Java actions into<br>
> "edp.java.main_class" and "edp.java.java_opts" configs.<br>
><br>
> I think yes.<br>
><br>
> Best,<br>
><br>
> Trevor<br>
><br>
> On Wed, 2014-01-29 at 09:15 -0500, Trevor McKay wrote:<br>
>> On Wed, 2014-01-29 at 14:35 +0400, Alexander Ignatov wrote:<br>
>>> Thank you for bringing this up, Trevor.<br>
>>><br>
>>> EDP gets more diverse and it's time to change its model.<br>
>>> I totally agree with your proposal, but one minor comment.<br>
>>> Instead of "savanna." prefix in job_configs wouldn't it be better to make it<br>
>>> as "edp."? I think "savanna." is too more wide word for this.<br>
>><br>
>> +1, brilliant. EDP is perfect. I was worried about the scope of<br>
>> "savanna." too.<br>
>><br>
>>> And one more bureaucratic thing... I see you already started implementing it [1],<br>
>>> and it is named and goes as new EDP workflow [2]. I think new bluprint should be<br>
>>> created for this feature to track all code changes as well as docs updates.<br>
>>> Docs I mean public Savanna docs about EDP, rest api docs and samples.<br>
>><br>
>> Absolutely, I can make it new blueprint. Thanks.<br>
>><br>
>>> [1] <a href="https://review.openstack.org/#/c/69712" target="_blank">https://review.openstack.org/#/c/69712</a><br>
>>> [2] <a href="https://blueprints.launchpad.net/openstack/?searchtext=edp-oozie-streaming-mapreduce" target="_blank">https://blueprints.launchpad.net/openstack/?searchtext=edp-oozie-streaming-mapreduce</a><br>
>>><br>
>>> Regards,<br>
>>> Alexander Ignatov<br>
>>><br>
>>><br>
>>><br>
>>> On 28 Jan 2014, at 20:47, Trevor McKay <<a href="mailto:tmckay@redhat.com">tmckay@redhat.com</a>> wrote:<br>
>>><br>
>>>> Hello all,<br>
>>>><br>
>>>> In our first pass at EDP, the model for job settings was very consistent<br>
>>>> across all of our job types. The execution-time settings fit into this<br>
>>>> (superset) structure:<br>
>>>><br>
>>>> job_configs = {'configs': {}, # config settings for oozie and hadoop<br>
>>>> 'params': {}, # substitution values for Pig/Hive<br>
>>>> 'args': []} # script args (Pig and Java actions)<br>
>>>><br>
>>>> But we have some things that don't fit (and probably more in the<br>
>>>> future):<br>
>>>><br>
>>>> 1) Java jobs have 'main_class' and 'java_opts' settings<br>
>>>> Currently these are handled as additional fields added to the<br>
>>>> structure above. These were the first to diverge.<br>
>>>><br>
>>>> 2) Streaming MapReduce (anticipated) requires mapper and reducer<br>
>>>> settings (different than the mapred.xxxx.class settings for<br>
>>>> non-streaming MapReduce)<br>
>>>><br>
>>>> Problems caused by adding fields<br>
>>>> --------------------------------<br>
>>>> The job_configs structure above is stored in the database. Each time we<br>
>>>> add a field to the structure above at the level of configs, params, and<br>
>>>> args, we force a change to the database tables, a migration script and a<br>
>>>> change to the JSON validation for the REST api.<br>
>>>><br>
>>>> We also cause a change for python-savannaclient and potentially other<br>
>>>> clients.<br>
>>>><br>
>>>> This kind of change seems bad.<br>
>>>><br>
>>>> Proposal: Borrow a page from Oozie and add "savanna." configs<br>
>>>> -------------------------------------------------------------<br>
>>>> I would like to fit divergent job settings into the structure we already<br>
>>>> have. One way to do this is to leverage the 'configs' dictionary. This<br>
>>>> dictionary primarily contains settings for hadoop, but there are a<br>
>>>> number of "oozie.xxx" settings that are passed to oozie as configs or<br>
>>>> set by oozie for the benefit of running apps.<br>
>>>><br>
>>>> What if we allow "savanna." settings to be added to configs? If we do<br>
>>>> that, any and all special configuration settings for specific job types<br>
>>>> or subtypes can be handled with no database changes and no api changes.<br>
>>>><br>
>>>> Downside<br>
>>>> --------<br>
>>>> Currently, all 'configs' are rendered in the generated oozie workflow.<br>
>>>> The "savanna." settings would be stripped out and processed by Savanna,<br>
>>>> thereby changing that behavior a bit (maybe not a big deal)<br>
>>>><br>
>>>> We would also be mixing "savanna." configs with config_hints for jobs,<br>
>>>> so users would potentially see "savanna.xxxx" settings mixed with oozie<br>
>>>> and hadoop settings. Again, maybe not a big deal, but it might blur the<br>
>>>> lines a little bit. Personally, I'm okay with this.<br>
>>>><br>
>>>> Slightly different<br>
>>>> ------------------<br>
>>>> We could also add a "'savanna-configs': {}" element to job_configs to<br>
>>>> keep the configuration spaces separate.<br>
>>>><br>
>>>> But, now we would have 'savanna-configs' (or another name), 'configs',<br>
>>>> 'params', and 'args'. Really? Just how many different types of values<br>
>>>> can we come up with? :)<br>
>>>><br>
>>>> I lean away from this approach.<br>
>>>><br>
>>>> Related: breaking up the superset<br>
>>>> ---------------------------------<br>
>>>><br>
>>>> It is also the case that not every job type has every value type.<br>
>>>><br>
>>>> Configs Params Args<br>
>>>> Hive Y Y N<br>
>>>> Pig Y Y Y<br>
>>>> MapReduce Y N N<br>
>>>> Java Y N Y<br>
>>>><br>
>>>> So do we make that explicit in the docs and enforce it in the api with<br>
>>>> errors?<br>
>>>><br>
>>>> Thoughts? I'm sure there are some :)<br>
>>>><br>
>>>> Best,<br>
>>>><br>
>>>> Trevor<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> OpenStack-dev mailing list<br>
>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> OpenStack-dev mailing list<br>
>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
CONFIDENTIALITY NOTICE<br>
NOTICE: This message is intended for the use of the individual or entity to<br>
which it is addressed and may contain information that is confidential,<br>
privileged and exempt from disclosure under applicable law. If the reader<br>
of this message is not the intended recipient, you are hereby notified that<br>
any printing, copying, dissemination, distribution, disclosure or<br>
forwarding of this communication is strictly prohibited. If you have<br>
received this communication in error, please contact the sender immediately<br>
and delete it from your system. Thank You.<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Sincerely yours,</div><div>Sergey Lukjanov</div><div>Savanna Technical Lead</div><div>Mirantis Inc.</div></div>
</div>