<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Hi,</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
I asked this question in Launchpad (<a href="https://answers.launchpad.net/savanna/+question/240969" style="font-family: Calibri, sans-serif; font-size: 14px; ">https://answers.launchpad.net/savanna/+question/240969</a>), but I thought it might reach more people
if I use the list.</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
My set up is:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Ubuntu 12.04<br>
OpenStack Havana with Vanilla Plugin</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
I have deployed a cluster with the following node groups:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
1 x master:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
-Uses 1 cinder volume : 2TB</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
-namenode<br>
-secondarynam<wbr>enode<br>
-oozie<br>
-datanode<br>
-jobtracker<br>
-tasktracker</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
2x slaves:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
-Uses 1 cinder volume: 2TB</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
-datanode<br>
-tasktracker</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Both node groups used the following flavor:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
VCPUs: 32<br>
RAM: 250000<br>
Root disk: 300GB<br>
Ephemeral: 300GB<br>
Swap: 0</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
They also use the default Ubuntu Hadoop Vanilla image downloadable from <a rel="nofollow" href="https://savanna.readthedocs.org/en/latest/userdoc/vanilla_plugin.html" style="color: rgb(0, 51, 170); text-decoration: none;">https:/<wbr>/savanna.<wbr>readthedocs.<wbr>org/en/<wbr>latest/<wbr>userdoc/<wbr>vanilla_<wbr>plugin.<wbr>html</a></p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
The /etc/hosts file in all nodes is:<br>
127.0.0.1 localhost<br>
10.0.0.2 test-master2T-<wbr>001.novalocal test-master2T-001<br>
10.0.0.3 test-slave2T-<wbr>001.novalocal test-slave2T-001<br>
10.0.0.4 test-slave2T-<wbr>002.novalocal test-slave2T-002</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Without changing any of the default configuration, the cluster boots correctly.</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
The problem is that, when running a job (for example, teragen 100GB), the map tasks fail many times, having to repeat them, thus increasing the job time. They seem to fail randomly, from one slave or the other, depending on the execution.</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Checking the logs of the datanotes in the slaves, I can see this error:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
WARN org.apache.<wbr>hadoop.<wbr>hdfs.server.<wbr>datanode.<wbr>DataNode: java.net.<wbr>ConnectExceptio<wbr>n: Call to test-master2T-<wbr>001/10.<wbr>0.0.2:8020 failed on connection exception: java.net.<wbr>ConnectExceptio<wbr>n: Connection refused</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Full error: <a rel="nofollow" href="http://pastebin.com/DDp39yqt" style="color: rgb(0, 51, 170); text-decoration: none;">http://<wbr>pastebin.<wbr>com/DDp39yqt</a></p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
The logs of the datanode in the master, gives this error:</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
WARN org.apache.<wbr>hadoop.<wbr>hdfs.server.<wbr>datanode.<wbr>DataNode: checkDiskError: exception:<br>
java.net.<wbr>SocketException<wbr>: Original Exception : java.io.<wbr>IOException: Connection reset by peer</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Full error: <a rel="nofollow" href="http://pastebin.com/NXYXELQX" style="color: rgb(0, 51, 170); text-decoration: none;">http://<wbr>pastebin.<wbr>com/NXYXELQX</a></p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
I have tried changing hadoop.tmp.dir to point to the 2TB cinder volume /volumes/<wbr>disk1/lib/<wbr>hadoop/<wbr>hdfs/tmp, but nothing changed.</p>
<p style="margin: 0px 0px 0.8em; padding: 0px; width: auto; max-width: 45em; color: rgb(51, 51, 51); font-family: Ubuntu, 'Bitstream Vera Sans', 'DejaVu Sans', Tahoma, sans-serif; font-size: 12px; line-height: 18px; text-align: left; background-color: rgb(255, 255, 255); ">
Thank you in advance.</p>
</div>
<div>
<div><br>
</div>
<div>Marc</div>
</div>
</body>
</html>