[openstack-dev] [sahara] Notes on developing Sahara Spark EDP to work with swift:// paths

Gil Vernik GILV at il.ibm.com
Thu Aug 28 16:31:21 UTC 2014


Hi,

In case this is helpful for you, this is the patch i submitted to Spark 
about Swift and Spark integration ( about to be merged )
https://github.com/apache/spark/pull/1010

I sent information about this patch to this mailing list about two months 
ago.

All the best,
Gil.





From:   Trevor McKay <tmckay at redhat.com>
To:     OpenStack Development Mailing List 
<openstack-dev at lists.openstack.org>
Date:   28/08/2014 06:22 PM
Subject:        [openstack-dev] [sahara] Notes on developing Sahara Spark 
EDP to work with swift:// paths



Hi folks,

  I've updated this etherpad with notes from an investigation of
Spark/Swift and the hadoop-openstack plugin carried in the sahara-extra
repo.
 
  Following the notes there, I was able to access swift:// paths from
Spark jobs on a Spark standalone cluster launched from Sahara and then
fixed up by hand.

  Comments welcome.  This is a POC at this point imho, we have work to
do to fully integrate this into Sahara.

https://etherpad.openstack.org/p/sahara_spark_edp

Best,

Trevor


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140828/e128f77e/attachment.html>


More information about the OpenStack-dev mailing list