Open Stack

Additionally, I think that we should explicitly specify the need to ensure
that all outputs doesn't contain any sensitive information like credentials.


On Tue, Jan 28, 2014 at 4:06 PM, Alexander Ignatov <aignatov at mirantis.com>wrote:

> "EDP internal" I meant current EDP specific code. And since job configs
> are job-specific
>  I'd prefer to have urls containing /jobs or at least /edp for that.
>
> Regards,
> Alexander Ignatov
>
>
>
> On 24 Jan 2014, at 23:20, Matthew Farrellee <matt at redhat.com> wrote:
>
> > what do you consider "EDP internal", and how does it relate to the v1.1
> or v2 API?
> >
> > i'm ok with making it plugin independent. i'd just suggest moving it out
> of /jobs and to something like /extra/config-hints/{type}, maybe along with
> /extra/validations/config.
> >
> > best,
> >
> >
> > matt
> >
> > On 01/22/2014 06:25 AM, Alexander Ignatov wrote:
> >> Current EDP config-hints are not only plugin specific. Several types of
> jobs
> >> must have certain key/values and without it job will fail. For instance,
> >> MapReduce (former Jar) job type requires Mapper/Reducer classes
> parameters
> >> to be set[1]. Moreover, for such kind of jobs we already have separated
> >> configuration defaults [2]. Also initial versions of patch implementing
> >> config-hints contained plugin-independent defaults for all each job
> types [3].
> >> I remember we postponed decision about which configs are commmon for all
> >> plugins and agreed to show users all vanilla-specific defaults. That's
> why now
> >> we have several TODOs in the code about config-hints should be
> plugin-specific.
> >>
> >> So I propose to leave config-hints REST call in EDP internal and make it
> >> plugin-independent (or job-specific) by removing of parsing all
> vanilla-specific
> >> defaults and define small list of configs which is definitely common
> for each type of jobs.
> >> The first things come to mind:
> >> - For MapReduce jobs it's already defined in [1]
> >> - Configs like number of map and reduce tasks are common for all type
> of jobs
> >> - At least user always has an ability to set any key/value(s) as
> params/arguments for job
> >>
> >>
> >> [1]
> http://docs.openstack.org/developer/savanna/userdoc/edp.html#workflow
> >> [2]
> https://github.com/openstack/savanna/blob/master/savanna/service/edp/resources/mapred-job-config.xml
> >> [3] https://review.openstack.org/#/c/45419/10
> >>
> >> Regards,
> >> Alexander Ignatov
> >>
> >>
> >>
> >> On 20 Jan 2014, at 22:04, Matthew Farrellee <matt at redhat.com> wrote:
> >>
> >>> On 01/20/2014 12:50 PM, Andrey Lazarev wrote:
> >>>> Inlined.
> >>>>
> >>>>
> >>>> On Mon, Jan 20, 2014 at 8:15 AM, Matthew Farrellee <matt at redhat.com
> >>>> <mailto:matt at redhat.com>> wrote:
> >>>>
> >>>>    (inline, trying to make this readable by a text-only mail client
> >>>>    that doesn't use tabs to indicate quoting)
> >>>>
> >>>>    On 01/20/2014 02:50 AM, Andrey Lazarev wrote:
> >>>>
> >>>>                 ------
> >>>>                 FIX - @rest.get('/jobs/config-hints/____<job_type>') -
> >>>>        should move to
> >>>>                 GET /plugins/<plugin_name>/<____plugin_version>,
> similar to
> >>>>                 get_node_processes
> >>>>                 and get_required_image_tags
> >>>>                 ------
> >>>>                 Not sure if it should be plugin specific right now.
> EDP
> >>>>        uses it
> >>>>                 to show some
> >>>>                 configs to users in the dashboard. it's just a
> cosmetic
> >>>>        thing.
> >>>>                 Also when user
> >>>>                 starts define some configs for some job he might not
> define
> >>>>                 cluster yet and
> >>>>                 thus plugin to run this job. I think we should leave
> it
> >>>>        as is
> >>>>                 and leave only
> >>>>                 abstract configs like Mapper/Reducer class and allow
> >>>>        users to
> >>>>                 apply any
> >>>>                 key/value configs if needed.
> >>>>
> >>>>
> >>>>             FYI, the code contains comments suggesting it should be
> >>>>        plugin specific.
> >>>>
> >>>>
> https://github.com/openstack/____savanna/blob/master/savanna/____service/edp/workflow_creator/____workflow_factory.py#L179
> >>>>        <
> https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179
> >
> >>>>
> >>>>        <
> https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179
> >>>>        <
> https://github.com/openstack/savanna/blob/master/savanna/service/edp/workflow_creator/workflow_factory.py#L179
> >>
> >>>>
> >>>>             IMHO, the EDP should have no plugin specific dependencies.
> >>>>
> >>>>             If it currently does, we should look into why and see if
> we
> >>>>        can't
> >>>>             eliminate this entirely.
> >>>>
> >>>>        [AL] EDP uses plugins in two ways:
> >>>>        1. for HDFS user
> >>>>        2. for config hints
> >>>>        I think both items should not be plugin specific on EDP API
> >>>>        level. But
> >>>>        implementation should go to plugin and call plugin API for
> result.
> >>>>
> >>>>
> >>>>    In fact they are both plugin specific. The user is forced to click
> >>>>    through a plugin selection (when launching a job on transient
> >>>>    cluster) or the plugin selection has already occurred (when
> >>>>    launching a job on an existing cluster).
> >>>>
> >>>>    Since the config is something that is plugin specific, you might
> not
> >>>>    have hbase hints from vanilla but you would from hdp, and you
> >>>>    already have plugin information whenever you ask for a hint, my
> view
> >>>>    that this be under the /plugins namespace is growing stronger.
> >>>>
> >>>>
> >>>> [AL] Disagree. They are plugin specific, but EDP itself could have
> >>>> additional plugin-independent logic inside. Now config hints return
> EDP
> >>>> properties (like mapred.input.dir) as well as plugin-specific
> >>>> properties. Placing it under /plugins namespace will give a vision
> that
> >>>> it is fully plugin specific.
> >>>>
> >>>> I like to see EDP API fully plugin independent and in one workspace.
> If
> >>>> core side needs some information internally it can easily go into the
> >>>> plugin.
> >>>
> >>> I'm not sure if we're disagreeing. We may, in fact, be in violent
> agreement.
> >>>
> >>> The EDP API is fully plugin independent, and should stay that way as a
> project goal. config-hints is extra data that the horizon app can use to
> help give users suggestions about what config they may want to optionally
> add to their job. Those config options are independent of the job and
> specific to the cluster where the job will run, which is the purview of the
> plugin.
> >>>
> >>> Moving config-hints out of the EDP API will make this even more clear.
> >>>
> >>> Best,
> >>>
> >>>
> >>> matt
> >>>
> >>> _______________________________________________
> >>> OpenStack-dev mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Sincerely yours,
Sergey Lukjanov
Savanna Technical Lead
Mirantis Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140128/0746d4c2/attachment.html>