[openstack-dev] [sahara] Notes on developing Sahara Spark EDP to work with swift:// paths

Michael McCune mimccune at redhat.com
Thu Aug 28 17:12:11 UTC 2014


hi Gil,

that's cool about the patch to Spark, has there been any talk about upgrading that patch to include Keystone v3 operations?

----- Original Message -----
> Hi,
> 
> In case this is helpful for you, this is the patch i submitted to Spark
> about Swift and Spark integration ( about to be merged )
> https://github.com/apache/spark/pull/1010
> 
> I sent information about this patch to this mailing list about two months
> ago.
> 
> All the best,
> Gil.
> 
> 
> 
> 
> 
> From:   Trevor McKay <tmckay at redhat.com>
> To:     OpenStack Development Mailing List
> <openstack-dev at lists.openstack.org>
> Date:   28/08/2014 06:22 PM
> Subject:        [openstack-dev] [sahara] Notes on developing Sahara Spark
> EDP to work with swift:// paths
> 
> 
> 
> Hi folks,
> 
>   I've updated this etherpad with notes from an investigation of
> Spark/Swift and the hadoop-openstack plugin carried in the sahara-extra
> repo.
>  
>   Following the notes there, I was able to access swift:// paths from
> Spark jobs on a Spark standalone cluster launched from Sahara and then
> fixed up by hand.
> 
>   Comments welcome.  This is a POC at this point imho, we have work to
> do to fully integrate this into Sahara.
> 
> https://etherpad.openstack.org/p/sahara_spark_edp
> 
> Best,
> 
> Trevor
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 



More information about the OpenStack-dev mailing list