<font face="verdana,sans-serif">I took a look at some distributed file
systems and went a little deeper in Hadoop and his HDFS, for instance. I
don't really need full POSIX compliance, but having a nested structure
is important, but as far as I know there are way to simulate this on
Switf, is that correct?<br>
<br>
</font>The problem I see in using something like hadoop is the single point of failure, not because I need almost 100% availability, but because the people who will access the data does not belong to the same organization. They will be researchers from different institutions that may want to deploy a local server with a subset of the data to improve their productivity, but the data set's size makes impractical to just copy everything. <br>
<br>The plan would be that the interface to the system would show which files are stored locally and which are not, so that everyone gets access to everything, almost like a peer to peer system where they download from the closest source and then store for their own use. <br>
<br>At first, I though of implementing something by hand, but using an already mature solution makes a lot more sense.<br><br>So, is this plausible or am I trying to use the wrong tools?<br><br>thanks, again<br><br>Thiago Moraes - EnC 07 - UFSCar<br>
<br><br><div class="gmail_quote">2011/8/14 Todd Deshane <span dir="ltr"><<a href="mailto:todd.deshane@xen.org">todd.deshane@xen.org</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
On Sun, Aug 14, 2011 at 4:10 AM, Thiago Moraes<br>
<div class="im"><<a href="mailto:thiago.camposmoraes@gmail.com">thiago.camposmoraes@gmail.com</a>> wrote:<br>
</div><div class="im">> Hey guys,<br>
><br>
> I'm new on the list and I'm currently considering Openstack to solve a data<br>
> distribution problem. Right now, there's a server which contains very large<br>
> files (usual files have 30GB or even more). This server is accessed by LAN<br>
> and over the internet but, of course, it's difficult to do this without<br>
> local connection.<br>
><br>
> My idea to solve this problem is to deploy new servers on the places which<br>
> access data more often in an such a way that they get a local copy of the<br>
> most accessed part of data by then. In my head, I consider that there will<br>
> be N different clouds, one at my location and the others spread on another<br>
> networks. Then, these new clouds would download and store parts of the data<br>
> (entire files) so that they can be accessed through their own LAN.<br>
><br>
<br>
</div>It sounds like you are looking for the functionality that Zones (aim<br>
to?) provide.<br>
<br>
Take a look at:<br>
<br>
<a href="http://wiki.openstack.org/MultiClusterZones" target="_blank">http://wiki.openstack.org/MultiClusterZones</a><br>
<div class="im"><br>
<br>
> Is Openstack suitable in this environment? Anyone would recommend another<br>
> solution?<br>
><br>
<br>
</div>Have you also looked at SheepDog, Hadoop or HC2? All of these seem to<br>
have some OpenStack integration points as well.<br>
<br>
Some links to look into:<br>
<a href="http://wiki.openstack.org/SheepdogSupport" target="_blank">http://wiki.openstack.org/SheepdogSupport</a><br>
<a href="http://doubleclix.wordpress.com/2011/03/17/hadoop-2-0-openstack-pbj/" target="_blank">http://doubleclix.wordpress.com/2011/03/17/hadoop-2-0-openstack-pbj/</a><br>
<a href="http://www.quora.com/What-features-differentiate-HDFS-and-OpenStack-Object-Storage" target="_blank">http://www.quora.com/What-features-differentiate-HDFS-and-OpenStack-Object-Storage</a><br>
<br>
<br>
Hope that helps.<br>
<br>
Thanks,<br>
Todd<br>
<div class="im"><br>
> PS: I know the file size limitations of 5GB. I just need that all parts of a<br>
> file to be in the same local area network so that a blazingly fast Internet<br>
> connection is not required all the time.<br>
><br>
> thanks,<br>
><br>
><br>
> Thiago Moraes - EnC 07 - UFSCar<br>
><br>
</div>> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
> Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
<font color="#888888"><br>
<br>
<br>
--<br>
Todd Deshane<br>
<a href="http://www.linkedin.com/in/deshantm" target="_blank">http://www.linkedin.com/in/deshantm</a><br>
<a href="http://www.xen.org/products/cloudxen.html" target="_blank">http://www.xen.org/products/cloudxen.html</a><br>
<a href="http://runningxen.com/" target="_blank">http://runningxen.com/</a><br>
</font></blockquote></div><br>