Open Stack

Answers inlined.


On Fri, Jan 10, 2014 at 1:05 PM, Daniele Venzano <Daniele.Venzano at eurecom.fr
> wrote:

> On 01/09/14 19:12, Matthew Farrellee wrote:
>
>> This is definitely great news!
>>
>> +2 to the things Sergey mentioned below.
>>
>> Additionally, will you fill out the blueprint or wiki w/ details that
>> will help others write integration tests for your plugin?
>>
>
> We already implemented at least some part of the integration tests for
> Spark, mimicking the ones that are provided with the Vanilla plugin. The
> Spark plugin works almost exactly as the Vanilla one, it can install a
> datanode, namenode, Spark master or Spark worker and resize the cluster.
> What kind of documentation is needed?


[SL] Are you installing HDFS too? I think that some docs about how your
plugin works and about the Spark's requirements will be great.


>
>
>
>  And, did you integrate (or have plans to integrate) Spark into the EDP
>> workflows in Horizon?
>>
>
> We would like to have that functionality. Currently we are limited by the
> lack of a Swift service in our cluster. We will have one test installation
> in a short while and then we will see. What is the status of the HDFS
> datasource? We are very interested in that, but I lost track of the
> development during the holidays.


Is it possible to run Spark workloads using Oozie? Here is the external
HDFS support change request - https://review.openstack.org/#/c/47828/.


>
>
>
>
>  On 01/09/2014 03:41 AM, Sergey Lukjanov wrote:
>>
>>> Hi,
>>>
>>> I'm really glad to here that!
>>>
>>> Answers inlined.
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Jan 9, 2014 at 11:33 AM, Daniele Venzano
>>> <Daniele.Venzano at eurecom.fr <mailto:Daniele.Venzano at eurecom.fr>> wrote:
>>>
>>>     Hello,
>>>
>>>     we are finishing up the development of the Spark plugin for Savanna.
>>>     In the next few days we will deploy it on an OpenStack cluster with
>>>     real users to iron out the last few things. Hopefully next week we
>>>     will put the code on a public github repository in beta status.
>>>
>>> [SL] Awesome! Could you, please, share some info this installation if
>>> possible? like OpenStack cluster version and size, Savanna version,
>>> expected Spark cluster sizes and lifecycle, etc.
>>>
>>>
>>>     You can find the blueprint here:
>>>     https://blueprints.launchpad.__net/savanna/+spec/spark-plugin
>>>     <https://blueprints.launchpad.net/savanna/+spec/spark-plugin>
>>>
>>>     There are two things we need to release, the VM image and the code
>>>     itself.
>>>     For the image we created one ourselves and for the code we used the
>>>     Vanilla plugin as a base.
>>>
>>> [SL] You can use diskimage-builder [0] to prepare such images, we're
>>> already using it for building images for vanilla plugin [1].
>>>
>>>
>>>     We feel that our work could be interesting for others and we would
>>>     like to see it integrated in Savanna. What is the best way to
>>> proceed?
>>>
>>> [SL] Absolutely, it's a very interesting tool for data processing. IMO
>>> the best way is to create a change request to savanna for code review
>>> and discussion in gerrit, it'll be really the most effective way to
>>> collaborate. As for the best way of integration with Savanna - we're
>>> expecting to see it in the openstack/savanna repo like vanilla, HDP and
>>> IDH (which will be landed soon) plugins.
>>>
>>>
>>>     We did not follow the Gerrit workflow until now because development
>>>     happened internally.
>>>     I will prepare the repo on github with git-review and reference the
>>>     blueprint in the commit. After that, do you prefer that I send
>>>     immediately the code for review or should I send a link here on the
>>>     mailing list first for some feedback/discussion?
>>>
>>> [SL] It'll be better to immediately send the code for review.
>>>
>>>
>>>     Thank you,
>>>     Daniele Venzano, Hoang Do and Vo Thanh Phuc
>>>
>>>     _________________________________________________
>>>     OpenStack-dev mailing list
>>>     OpenStack-dev at lists.openstack.__org
>>>     <mailto:OpenStack-dev at lists.openstack.org>
>>>
>>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>>
>>> [0] https://github.com/openstack/diskimage-builder
>>> [1] https://github.com/openstack/savanna-image-elements
>>>
>>> Please, feel free to ping me if some help needed with gerrit or savanna
>>> internals stuff.
>>>
>>> Thanks.
>>>
>>> --
>>> Sincerely yours,
>>> Sergey Lukjanov
>>> Savanna Technical Lead
>>> Mirantis Inc.
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Sincerely yours,
Sergey Lukjanov
Savanna Technical Lead
Mirantis Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140110/b31b035c/attachment.html>