<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi Joe,</div><div><br></div><div>thanks for your comments. </div><div><br></div><div>We are considering to use Heat as infrastructure provisioning for future iterations. You are right, now we have basic working version of Savanna, but it’s only the provisioning part that is the base for the main goal - implement Elastic Data Processing that isn’t connected to infrastructure provisioning and requires long-running tasks executions and potentially complex workflows. Current team bandwidth doesn’t allow us to contribute into Heat, but I hope that we’ll be able to investigate it better and start contributing missed features step by step. We are planning to spend some time investigating Heat status as soon as have some of the bandwidth free up or somebody new will join Savanna willing to contribute in this direction.</div><div>
<br>Sincerely yours,<br>Sergey Lukjanov<br>Savanna Technical Lead<br>Mirantis Inc.<br>
</div>
<br><div><div>On Jul 30, 2013, at 23:33, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 30, 2013 at 3:56 AM, Ruslan Kamaldinov <span dir="ltr"><<a href="mailto:rkamaldinov@mirantis.com" target="_blank">rkamaldinov@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This question was asked several times. So we decided to provide a<br>
detailed response.<br>
<br>
1. The first question is “Why doesn’t Savanna use Heat to provision VMs?”<br>
<br>
Generally using Heat underneath for infrastructure provisioning looks<br>
reasonable. In a tactic perspective there are few factors making Heat<br>
usage underneath Savanna problematic:<br>
* Heat stability for Grizzly release. Savanna currently maintains<br>
Grizzly+ compatibility.<br></blockquote><div><br></div><div>This shouldn't be an issue anymore. Now that we are in Havana</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Installation of large Hadoop clusters (100+ nodes). Will be<br>
addressed by proposed architecture changes.<br></blockquote><div><br></div><div>What about heat doesn't work for this?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Anti-affinity support for HDFS redundancy in cloud environment<br></blockquote><div> </div><div>Why can't heat do this</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Circular dependencies - we should generate ‘/etc/hosts’ for all<br>
instances in provisioned cluster. We can’t use cloud init for this<br>
directly. There are a couple possible solutions using Heat, but none<br>
of them looks like a straightforward solution.<br></blockquote><div><br></div><div>So why not make heat do this better</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Level of complexity. We try to keep things as simple as possible.<br>
Adding extra layer will increase overall complexity of the solution.<br>
In addition both Savanna and Heat under active development changing<br>
lots of internals and even APIs and will require extra effort to<br>
coordinate.<br></blockquote><div><br></div><div>But coordinating will result in less duplicate work</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Here is what we’ll do:<br>
* Create a wiki page with text from this email<br>
* Create a list of requirements for Heat<br>
<br>
Once Heat fulfills all the requirements we will be able and should use<br>
Heat for VM provisioning.<br>
<br></blockquote><div><br></div><div>Now that you have a basic working version of Savana, I would prefer to see you help make heat fit work for you instead of reimplementing a subset of what it is trying to do. </div><div>
<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2. Let’s answer the second question - why we need Savanna? Can’t we<br>
use Heat to do what Savanna does?<br>
<br>
* Savanna provides bunch of Hadoop-specific features. It’ll be hard to<br>
provide them as Heat plugin<br>
* Savanna provides Hadoop-specific APIs and functionality. Heat use<br>
cases are mostly around provisioning/deployment.<br>
* Savanna provides integration with various Hadoop distributions<br>
through pluggable mechanism<br>
<br>
Now, more details on each item.<br>
Hadoop specific features:<br>
* Tight Swift integration. Hadoop can read and write from/to Swift<br>
object storage. Savanna provides required configs for the Hadoop<br>
cluster.<br>
* Usage of anti-affinity to preserve data-redundancy of HDFS nodes<br>
<br>
Hadoop-specific APIs and functionality:<br>
* Hadoop cluster scaling<br>
* Elastic Data Processing: <a href="https://wiki.openstack.org/wiki/Savanna/EDP" target="_blank">https://wiki.openstack.org/wiki/Savanna/EDP</a><br>
<br>
Integration with Hadoop distributions through pluggable mechanism:<br>
- Usually Hadoop cluster deployment is a multi-step operation. First<br>
step is to install management console (for instance Apache Ambari).<br>
Second step is to communicate with management console through REST API<br>
to provision Hadoop on the cluster. Savanna wraps all this operations<br>
under well-defined API.<br>
<br>
I hope all the items above explain why we need Savanna as a separate<br>
OpenStack service.<br>
<br>
<br>
3. Why can’t Savanna be used as a plugin for Heat?<br>
It should be and it will be someday.<br>
<br>
<br>
Regards,<br>
Ruslan<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Jul 25, 2013 at 7:42 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:<br>
><br>
> On Jul 23, 2013 12:34 PM, "Sergey Lukjanov" <<a href="mailto:slukjanov@mirantis.com">slukjanov@mirantis.com</a>> wrote:<br>
>><br>
>> Hi evereyone,<br>
>><br>
>> We’ve started working on upgrading Savanna architecture in version 0.3 to<br>
>> make it horizontally scalable.<br>
>><br>
>> The most part of information is in the wiki page -<br>
>> <a href="https://wiki.openstack.org/wiki/Savanna/NextGenArchitecture" target="_blank">https://wiki.openstack.org/wiki/Savanna/NextGenArchitecture</a>.<br>
>><br>
>> Additionally there are several blueprints created for this activity -<br>
>> <a href="https://blueprints.launchpad.net/savanna?searchtext=ng-" target="_blank">https://blueprints.launchpad.net/savanna?searchtext=ng-</a><br>
>><br>
>> We are looking for comments / questions / suggestions.<br>
><br>
> This sounds like most of this can be built around Heat, except maybe the<br>
> rest api to hadoop. So why not use heat for the deploy part?<br>
><br>
>><br>
>> P.S. The another thing that we’re working on in Savanna 0.3 is EDP<br>
>> (Elastic Data Processing).<br>
>><br>
>> Thank you!<br>
>><br>
>> Sincerely yours,<br>
>> Sergey Lukjanov<br>
>> Savanna Technical Lead<br>
>> Mirantis Inc.<br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>
_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></blockquote></div><br></body></html>