<div dir="ltr">Answers inlined.<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 10, 2014 at 1:05 PM, Daniele Venzano <span dir="ltr"><<a href="mailto:Daniele.Venzano@eurecom.fr" target="_blank">Daniele.Venzano@eurecom.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 01/09/14 19:12, Matthew Farrellee wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
This is definitely great news!<br>
<br>
+2 to the things Sergey mentioned below.<br>
<br>
Additionally, will you fill out the blueprint or wiki w/ details that<br>
will help others write integration tests for your plugin?<br>
</blockquote>
<br></div>
We already implemented at least some part of the integration tests for Spark, mimicking the ones that are provided with the Vanilla plugin. The Spark plugin works almost exactly as the Vanilla one, it can install a datanode, namenode, Spark master or Spark worker and resize the cluster.<br>
What kind of documentation is needed?</blockquote><div><br></div><div>[SL] Are you installing HDFS too? I think that some docs about how your plugin works and about the Spark's requirements will be great.</div><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
And, did you integrate (or have plans to integrate) Spark into the EDP<br>
workflows in Horizon?<br>
</blockquote>
<br></div>
We would like to have that functionality. Currently we are limited by the lack of a Swift service in our cluster. We will have one test installation in a short while and then we will see. What is the status of the HDFS datasource? We are very interested in that, but I lost track of the development during the holidays.</blockquote>
<div><br></div><div>Is it possible to run Spark workloads using Oozie? Here is the external HDFS support change request - <a href="https://review.openstack.org/#/c/47828/">https://review.openstack.org/#/c/47828/</a>.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5"><br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On 01/09/2014 03:41 AM, Sergey Lukjanov wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hi,<br>
<br>
I'm really glad to here that!<br>
<br>
Answers inlined.<br>
<br>
Thanks.<br>
<br>
<br>
On Thu, Jan 9, 2014 at 11:33 AM, Daniele Venzano<br>
<<a href="mailto:Daniele.Venzano@eurecom.fr" target="_blank">Daniele.Venzano@eurecom.fr</a> <mailto:<a href="mailto:Daniele.Venzano@eurecom.fr" target="_blank">Daniele.Venzano@<u></u>eurecom.fr</a>>> wrote:<br>
<br>
Hello,<br>
<br>
we are finishing up the development of the Spark plugin for Savanna.<br>
In the next few days we will deploy it on an OpenStack cluster with<br>
real users to iron out the last few things. Hopefully next week we<br>
will put the code on a public github repository in beta status.<br>
<br>
[SL] Awesome! Could you, please, share some info this installation if<br>
possible? like OpenStack cluster version and size, Savanna version,<br>
expected Spark cluster sizes and lifecycle, etc.<br>
<br>
<br>
You can find the blueprint here:<br>
<a href="https://blueprints.launchpad." target="_blank">https://blueprints.launchpad.</a>_<u></u>_net/savanna/+spec/spark-<u></u>plugin<br>
<<a href="https://blueprints.launchpad.net/savanna/+spec/spark-plugin" target="_blank">https://blueprints.launchpad.<u></u>net/savanna/+spec/spark-plugin</a><u></u>><br>
<br>
There are two things we need to release, the VM image and the code<br>
itself.<br>
For the image we created one ourselves and for the code we used the<br>
Vanilla plugin as a base.<br>
<br>
[SL] You can use diskimage-builder [0] to prepare such images, we're<br>
already using it for building images for vanilla plugin [1].<br>
<br>
<br>
We feel that our work could be interesting for others and we would<br>
like to see it integrated in Savanna. What is the best way to<br>
proceed?<br>
<br>
[SL] Absolutely, it's a very interesting tool for data processing. IMO<br>
the best way is to create a change request to savanna for code review<br>
and discussion in gerrit, it'll be really the most effective way to<br>
collaborate. As for the best way of integration with Savanna - we're<br>
expecting to see it in the openstack/savanna repo like vanilla, HDP and<br>
IDH (which will be landed soon) plugins.<br>
<br>
<br>
We did not follow the Gerrit workflow until now because development<br>
happened internally.<br>
I will prepare the repo on github with git-review and reference the<br>
blueprint in the commit. After that, do you prefer that I send<br>
immediately the code for review or should I send a link here on the<br>
mailing list first for some feedback/discussion?<br>
<br>
[SL] It'll be better to immediately send the code for review.<br>
<br>
<br>
Thank you,<br>
Daniele Venzano, Hoang Do and Vo Thanh Phuc<br>
<br>
______________________________<u></u>___________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.<u></u>__org<br>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<br>
<a href="http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev" target="_blank">http://lists.openstack.org/__<u></u>cgi-bin/mailman/listinfo/__<u></u>openstack-dev</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a>><br>
<br>
<br>
<br>
[0] <a href="https://github.com/openstack/diskimage-builder" target="_blank">https://github.com/openstack/<u></u>diskimage-builder</a><br>
[1] <a href="https://github.com/openstack/savanna-image-elements" target="_blank">https://github.com/openstack/<u></u>savanna-image-elements</a><br>
<br>
Please, feel free to ping me if some help needed with gerrit or savanna<br>
internals stuff.<br>
<br>
Thanks.<br>
<br>
--<br>
Sincerely yours,<br>
Sergey Lukjanov<br>
Savanna Technical Lead<br>
Mirantis Inc.<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Sincerely yours,</div><div>Sergey Lukjanov</div><div>Savanna Technical Lead</div><div>Mirantis Inc.</div></div>
</div></div>