<div dir="ltr"><div>> 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release <br><br>We create lrzip archive with compression level "2" (from 1-9 range, default is 7). So we don't have much potential for improvement here. Here are some tests:<br>
<br>== Bare metal 12 cpus, 32 G RAM ==<br>level 1 decompression total time: 00:00:27.70<br>level 2 decompression total time: 00:00:29.64<br><br>== Virtual server 1 cpu, 1G RAM ==<br>level 1 decompression total time: 00:05:17.12<br>
level 2 decompression total time: 00:05:22.86<br><br><br>I did some further research on this matter, increasing RAM to 4G did not help much as well:<br><br>== Virtual server 1 cpu, 4G RAM ==<br>level 1 decompression total time: 00:05:08.70<br>
level 2 decompression total time: 00:05:12.57<br><br>So it looks like 'lrzuntar' command itself causes this problem on a weak hardware, because it uses "lrzcat | tar" which works very fast if we have enough memory. If we extract archive in two separate steps (lrzip -d && tar -xf) instead of single "lrzuntar", then they take only ~1m30s summary (comparing to 5+ minutes with single lrzuntar command). Here are some test results for lrzip+tar commands:<br>
<br>== Bare metal 12 cpus, 32 G RAM ==<br>level 2 decompression total time: 00:00:50.12<br><br>== Virtual server 1 cpu, 1G RAM ==<br>level 2 decompression total time: 00:01:35.95<br><br>So "lrzuntar" works faster on a powerful hardware, lrzip+tar works faster on a weak hardware (like VMs).<br>
<br></div>I suggest to switch to lrzip+tar.<br><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 10, 2014 at 8:01 PM, Dmitry Borodaenko <span dir="ltr"><<a href="mailto:dborodaenko@mirantis.com" target="_blank">dborodaenko@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">FWIW 1GB works fine for me on my laptop, I run the master setup manually. So I'm against increasing RAM requirement, we have better things to spend that RAM on.</p>
<div class="HOEnZb"><div class="h5">
<div class="gmail_quote">On May 10, 2014 1:37 AM, "Mike Scherbakov" <<a href="mailto:mscherbakov@mirantis.com" target="_blank">mscherbakov@mirantis.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">It is not related to RAM or CPU. I run installation on my Mac with 1Gb of RAM for master node, and experience the following:<div><ul><li>yes, it needs time to bootstrap admin node</li><li>As soon as I have message that master node is installed, I immediately open <a href="http://10.20.0.2:8000" target="_blank">10.20.0.2:8000</a> and try to generate diag snapshot. And it is failed.</li>
<li>If I wait a few more minutes, and try again - it is passed.</li></ul><div>It actually seems to me that we simply still do not have <a href="https://bugs.launchpad.net/fuel/+bug/1315865" target="_blank">https://bugs.launchpad.net/fuel/+bug/1315865</a> fixed, I'll add more details there as well as logs.</div>
</div><div><br></div><div>When I checked logs, I saw:</div><div><ul><li>for about a minute, astute was not able to connect to MQ. It means it is still started before MQ is ready?</li><li><font face="courier new, monospace">shotgun -c /tmp/dump_config >> /var/log/dump.log 2>&1 && cat /var/www/nailgun/dump/last </font><font face="arial, helvetica, sans-serif">returned 1</font></li>
</ul>When I tried to run diag_snapshot for a second time, the command above succeeded with 0 return code.</div><div><br></div><div>So it obviously needs further debugging and in my opinion even if we need to increase VCPU or RAM, then no more than 2 VCPU / 2 Gb.<br>
</div><div><br></div><div>Vladimir, as you and Matt were changing the code which should run containers in a certain order, I'm looking forward to hear from both of you suggestions on where and how we should hack it.</div>
<div><br></div><div>Thanks,</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 10, 2014 at 1:04 AM, Vladimir Kuklin <span dir="ltr"><<a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all<div><br></div><div>We are still experiencing some issues with master node bootstrapping after moving to container-based installation.</div>
<div><br></div><div>First of all, these issues are related to our system tests. We have rather small nodes running as master node - only 1 GB of RAM and 1 virtual CPU. As we are using strongly lrzipped archive, this seems quite not enough and leads to timeouts during deployment of the master node.</div>
<div><br></div><div>I have several suggestions:</div><div><br></div><div>1) Increase amount of RAM for master node to at least 8 Gigabytes (or do some pci virtual memory hotplug during master node bootstrapping) and add additional vCPU for the master node.</div>
<div>2) Run system tests with non-containerized environment (env variable PRODUCTION=prod set)</div><div>3) Split our system tests in that way not allowing more than 2 master nodes to bootstrap simulteneously on the single hardware node.</div>
<div>4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release </div><div>5) increase bootstrap timeout for the master node in system tests</div><div><br></div><div><br>
</div><div>Any input would be appreciated.</div><div><div><br></div>-- <br><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br><a href="tel:%2B7%20%28495%29%20640-49-04" value="+74956404904" target="_blank">+7 (495) 640-49-04</a><br>
<a href="tel:%2B7%20%28926%29%20702-39-68" value="+79267023968" target="_blank">+7 (926) 702-39-68</a><br>
Skype kuklinvv<br>
45bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div>
</div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Mike Scherbakov<br>#mihgen<br><br>
</div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>