<div dir="ltr">Dear Alejandro,<br><br>every time we run a script to prune the _base directory in our enviroments we raise a pray to Santa Bárbara (<a href="http://es.wikipedia.org/wiki/B%C3%A1rbara_(m%C3%A1rtir)#Patronazgo">http://es.wikipedia.org/wiki/B%C3%A1rbara_(m%C3%A1rtir)#Patronazgo</a>). Yes, _base folder with shared storage is a risky business.<div>
<br></div><div>Have you considered not using _base backed images? This article can help: <a href="http://www.pixelbeat.org/docs/openstack_libvirt_images/">http://www.pixelbeat.org/docs/openstack_libvirt_images/</a></div>
<div>
<br></div><div>If I'm right and you are using some kind of NFS device with deduplication, may be it can be an option.</div><div><br></div><div>Cheers</div><div>Diego</div><div><br></div><div><br></div><div><br></div>
<div>
<br></div><div><br></div><img class="mailtrack-img" src="https://mailtrack.io/trace/mail/8ee51757c2acc92b5c7805ce5c3f81d3f4e442c5.png" height="1" width="1"></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">
-- <br><span style="border-collapse:separate;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif"><div align="left" style="font-size:13px"><div><font><span lang="ES" style="font-family:Arial">Diego Parrilla<br>
<a href="http://www.stackops.com/" title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com
www.garrigues.com" style="color:rgb(7,77,143)" target="_blank"><span title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com"></span></a></span></font><font face="Arial" style="font-family:arial,helvetica,sans-serif"><b>CEO</b><font size="1"><br>
</font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"></span></span><b><font face="Arial" style="font-family:arial,helvetica,sans-serif"><a href="http://www.stackops.com/" target="_blank"><b>www.stackops.com</b></a> | </font></b><font face="Arial" style="font-family:arial,helvetica,sans-serif"><font size="1"> <a href="mailto:diego.parrilla@stackops.com" target="_blank">diego.parrilla@stackops.com</a></font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span lang="EN-GB" style="font-size:10pt"></span></b></b></font></span></span><font face="Arial" style="font-family:arial,helvetica,sans-serif"><font size="1"> | </font></font><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">+34 91 005-2164 | </span><a style="font-size:x-small;font-family:arial,helvetica,sans-serif">skype:diegoparrilla</a></div>
</div><div style="font-size:13px"><font color="#004438" face="Arial"><b><p><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span lang="EN-GB" style="font-size:10pt"><img src="http://stackops.s3-external-3.amazonaws.com/STACKOPSLOGO-ICON.png"></span></b></b></font></span></span></span></span></p>
</b></font></div></span></span><div><br></div></div></div>
<br><br><div class="gmail_quote">On Wed, Apr 2, 2014 at 8:04 PM, Alejandro Comisario <span dir="ltr"><<a href="mailto:alejandro.comisario@mercadolibre.com" target="_blank">alejandro.comisario@mercadolibre.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi guys ...<br>
We have a pretty big openstack environment and we use a shared NFS to<br>
populate backing file directory ( the famous _base directory located<br>
on /var/lib/nova/instances/_base ) due to a human error, the backing<br>
file used by thousands of guests was deleted, causing this guests to<br>
go read-only filesystem in a second.<br>
<br>
Till that moment we were convinced to use the _base directory as a<br>
shared NFS because:<br>
<br>
* spawning a new ami gives total visibility to the whole cloud making<br>
instances take nothing to boot despite the nova region<br>
* ease glance workload<br>
* easiest management no having to replicate files constantly not<br>
pushing bandwidth usage internally<br>
<br>
But after this really big issue, and after what took us to recover<br>
from this, we were thinking about how to protect against this kind of<br>
"single point of failure".<br>
Our first aproach this days was to put Read Only the NFS share, making<br>
impossible for computes ( and humans ) to write to that directory,<br>
giving permision to just one compute whos the one responsible to spawn<br>
an instance from a new ami and write the file to the directory, still<br>
... the storage keeps being the SPOF.<br>
<br>
So, we are handling the possibility of having the used backing files<br>
LOCAL on every compute ( +1K hosts ) and reduce the failure chances to<br>
the minimum, obviously, with a pararell talk about what technology to<br>
use to keep data replicated among computes when a new ami is launched,<br>
launching times, performance matters on compute nodes having to store<br>
backing files locally, etc.<br>
<br>
This make me realize, i have a huge comminity behind openstack, so<br>
wanted to ear from it:<br>
<br>
* what are your thoughts about what happened / what we are thinking right now ?<br>
* how does other users manage the backing file ( _base ) directory<br>
having all this considerations on big openstack deployments ?<br>
<br>
I will be thrilled to read from other users, experiences and thoughts.<br>
<br>
As allways, best.<br>
Alejandro<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote></div><br></div>