[openstack-dev] [sahara] summit wrap-up: Future of EDP

Sergey Lukjanov slukjanov at mirantis.com
Tue Jun 3 17:46:45 UTC 2014

Cool, thank you for wrapping up design summit session!

On Tue, Jun 3, 2014 at 7:24 PM, Trevor McKay <tmckay at redhat.com> wrote:
> Hi folks,
>   Here is a summary of priorities from Summit and some action items for
> high the high priority issues.  The link to the pad is here:
> https://etherpad.openstack.org/p/juno-summit-sahara-edp
>   We really did not have any leftover questions from summit, but we need
> investigation and development work in several areas.  Please respond with any
> comments/questions, and feel free to work on action items :)
> High priority
>     Fix hive support
>     minimal EDP for spark via spark plugin (may be possible w Oozie)
>     Design pluggable job model and investigate Spark / Storm integration
> Medium priority
>     Error reporting improvements
> Low priority
>     Raw Oozie workflows
>     coordinated jobs
>     preparation tags for workflows
>     files and archives tags (need clear use cases)
>     streamline copying of job binaries from swift to hdfs (dscp)
> Action items for high priority issues:
>     Hive:
>         We need to flesh out existing blueprints:
>             https://blueprints.launchpad.net/sahara/+spec/hive-vanilla2-support
>             https://blueprints.launchpad.net/sahara/+spec/hive-integration-tests
>         Additional blueprint needed for swift support in Hive
>         Hive is not fully implemented in HDP plugin
>     Investigate Spark job execution via Oozie
>         Underway.  Produce a blueprint after initial investigation.
>     Design a pluggable job model
>         We need to review current EDP and think about how operations can be
>         abstracted.  For instance, what are the essential operations on a job,
>         and where has the Oozie implementation leaked knowledge into the EDP code?
>         Can we develop an abstraction that maps to other models in a believable way?
>            storm, spark, scalding, others
>         How will the UI deal with a pluggable job model assuming we have one?
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Sincerely yours,
Sergey Lukjanov
Sahara Technical Lead
(OpenStack Data Processing)
Principal Software Engineer
Mirantis Inc.

More information about the OpenStack-dev mailing list