Open Stack

“EDP internal” I meant current EDP specific code. And since job configs are job-specific
 I’d prefer to have urls containing /jobs or at least /edp for that.

Regards,
Alexander Ignatov



On 24 Jan 2014, at 23:20, Matthew Farrellee <matt at redhat.com> wrote:

> what do you consider "EDP internal", and how does it relate to the v1.1 or v2 API?
> 
> i'm ok with making it plugin independent. i'd just suggest moving it out of /jobs and to something like /extra/config-hints/{type}, maybe along with /extra/validations/config.
> 
> best,
> 
> 
> matt
> 
> On 01/22/2014 06:25 AM, Alexander Ignatov wrote:
>> Current EDP config-hints are not only plugin specific. Several types of jobs
>> must have certain key/values and without it job will fail. For instance,
>> MapReduce (former Jar) job type requires Mapper/Reducer classes parameters
>> to be set[1]. Moreover, for such kind of jobs we already have separated
>> configuration defaults [2]. Also initial versions of patch implementing
>> config-hints contained plugin-independent defaults for all each job types [3].
>> I remember we postponed decision about which configs are commmon for all
>> plugins and agreed to show users all vanilla-specific defaults. That's why now
>> we have several TODOs in the code about config-hints should be plugin-specific.
>> 
>> So I propose to leave config-hints REST call in EDP internal and make it
>> plugin-independent (or job-specific) by removing of parsing all vanilla-specific
>> defaults and define small list of configs which is definitely common for each type of jobs.
>> The first things come to mind:
>> - For MapReduce jobs it's already defined in [1]
>> - Configs like number of map and reduce tasks are common for all type of jobs
>> - At least user always has an ability to set any key/value(s) as params/arguments for job
>> 
>> 
>> [1] http://docs.openstack.org/developer/savanna/userdoc/edp.html#workflow
>> [2] https://github.com/openstack/savanna/blob/master/savanna/service/edp/resources/mapred-job-config.xml
>> [3] https://review.openstack.org/#/c/45419/10
>> 
>> Regards,
>> Alexander Ignatov
>> 
>> 
>> 
>> On 20 Jan 2014, at 22:04, Matthew Farrellee <matt at redhat.com> wrote:
>> 
>>> On 01/20/2014 12:50 PM, Andrey Lazarev wrote:
>>>> Inlined.
>>>> 
>>>> 
>>>> On Mon, Jan 20, 2014 at 8:15 AM, Matthew Farrellee <matt at redhat.com
>>>> <mailto:matt at redhat.com>> wrote:
>>>> 
>>>>    (inline, trying to make this readable by a text-only mail client
>>>>    that doesn't use tabs to indicate quoting)
>>>> 
>>>>    On 01/20/2014 02:50 AM, Andrey Lazarev wrote:
>>>> 
>>>>                 ------
>>>>                 FIX - @rest.get('/jobs/config-hints/____<job_type>') -
>>>>        should move to
>>>>                 GET /plugins/<plugin_name>/<____plugin_version>, similar to
>>>>                 get_node_processes
>>>>                 and get_required_image_tags
>>>>                 ------
>>>>                 Not sure if it should be plugin specific right now. EDP
>>>>        uses it
>>>>                 to show some
>>>>                 configs to users in the dashboard. it's just a cosmetic
>>>>        thing.
>>>>                 Also when user
>>>>                 starts define some configs for some job he might not define
>>>>                 cluster yet and
>>>>                 thus plugin to run this job. I think we should leave it
>>>>        as is
>>>>                 and leave only
>>>>                 abstract configs like Mapper/Reducer class and allow
>>>>        users to
>>>>                 apply any
>>>>                 key/value configs if needed.
>>>> 
>>>> 
>>>>             FYI, the code contains comments suggesting it should be
>>>>        plugin specific.
>>>> 
>>>>        https://github.com/openstack/____savanna/blob/master/savanna/____service/edp/workflow_creator/____workflow_factory.py#L179
>>>>        <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179>
>>>> 
>>>>        <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179
>>>>        <https://github.com/openstack/savanna/blob/master/savanna/service/edp/workflow_creator/workflow_factory.py#L179>>
>>>> 
>>>>             IMHO, the EDP should have no plugin specific dependencies.
>>>> 
>>>>             If it currently does, we should look into why and see if we
>>>>        can't
>>>>             eliminate this entirely.
>>>> 
>>>>        [AL] EDP uses plugins in two ways:
>>>>        1. for HDFS user
>>>>        2. for config hints
>>>>        I think both items should not be plugin specific on EDP API
>>>>        level. But
>>>>        implementation should go to plugin and call plugin API for result.
>>>> 
>>>> 
>>>>    In fact they are both plugin specific. The user is forced to click
>>>>    through a plugin selection (when launching a job on transient
>>>>    cluster) or the plugin selection has already occurred (when
>>>>    launching a job on an existing cluster).
>>>> 
>>>>    Since the config is something that is plugin specific, you might not
>>>>    have hbase hints from vanilla but you would from hdp, and you
>>>>    already have plugin information whenever you ask for a hint, my view
>>>>    that this be under the /plugins namespace is growing stronger.
>>>> 
>>>> 
>>>> [AL] Disagree. They are plugin specific, but EDP itself could have
>>>> additional plugin-independent logic inside. Now config hints return EDP
>>>> properties (like mapred.input.dir) as well as plugin-specific
>>>> properties. Placing it under /plugins namespace will give a vision that
>>>> it is fully plugin specific.
>>>> 
>>>> I like to see EDP API fully plugin independent and in one workspace. If
>>>> core side needs some information internally it can easily go into the
>>>> plugin.
>>> 
>>> I'm not sure if we're disagreeing. We may, in fact, be in violent agreement.
>>> 
>>> The EDP API is fully plugin independent, and should stay that way as a project goal. config-hints is extra data that the horizon app can use to help give users suggestions about what config they may want to optionally add to their job. Those config options are independent of the job and specific to the cluster where the job will run, which is the purview of the plugin.
>>> 
>>> Moving config-hints out of the EDP API will make this even more clear.
>>> 
>>> Best,
>>> 
>>> 
>>> matt
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev