XFS keeps some useful runtime statistics on /proc/fs/xfs/stat that can be maybe used to test this theory:
<a href="http://xfs.org/index.php/Runtime_Stats">http://xfs.org/index.php/Runtime_Stats</a> <div><br></div><div>In particular, the following stats could be useful:</div><div><br></div><div><span style="font-family:sans-serif;font-size:12.800000190734863px;line-height:18.399999618530273px;background-color:rgb(255,255,255)">xs_dir_lookup (xfs.dir_ops.lookup)</span><ul style="line-height:18.399999618530273px;list-style-type:square;margin:0.3em 0px 0px 1.6em;padding:0px;font-family:sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
<li style="margin-bottom:0.1em">This is a count of the number of file name directory lookups in XFS filesystems. It counts only those lookups which miss in the operating system's directory name lookup cache and must search the real directory structure for the name in question. The count is incremented once for each level of a pathname search that results in a directory lookup.</li>
</ul><div><span style="font-family:sans-serif;font-size:12.800000190734863px;line-height:18.399999618530273px;background-color:rgb(255,255,255)">xs_ig_missed (xfs.inode_ops.ig_missed)</span><ul style="line-height:18.399999618530273px;list-style-type:square;margin:0.3em 0px 0px 1.6em;padding:0px;font-family:sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
<li style="margin-bottom:0.1em">This is the number of times the operating system looked for an XFS inode in the inode cache and the inode was not there. The further this count is from the ig_attempts count the better.</li>
</ul></div></div><div><br></div><div>Compare the number of cache misses when the system is normal, and when the system performance is degraded.<br><br>I've been running into similar problems too, in a setup with lots of small files. Does anyone knows how big is the XFS inode and directory caches?<br>
<br><div class="gmail_quote">2012/7/25 Robert van Leeuwen <span dir="ltr"><<a href="mailto:Robert.vanLeeuwen@spilgames.com" target="_blank">Robert.vanLeeuwen@spilgames.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma"><div class="im">>Last week , I got more servers from another HW providers with more CPU/RAM/DISKs . 12 Disks in >each storage node. This deployment of swift cluster keep in better performance
for longer time. >Unfortunately , after 15,000,000 object . The performance reduced to half and the Failure appeared.
</div><div style="font-size:16px;font-family:Times New Roman">
<div>
<div class="gmail_quote"><div class="im">
<div>>I concerned about that if the (total number objs/disk numbers) = ? will cause such affect in large >deployment.(aka. cloud storage provider , telecom , bank etc.) </div>
<br></div>
We also see lower than expected performance and we have many files on the nodes.<br>
We currently have about 15 million files per file-system/disk for our object servers with 6 disks per machine (90 Million files on one node) and 48GB of memory.
<br>
<br>
The object-servers are io-bound when we do put's, effectively doing about 1 write per disk we have in the cluster.
<br>
(while also doing about 2 reads per disk at the same time)<br>
This is way lower than expected, especially since we also use flashcache with 10GB caching per disk and the container nodes are on seperate hardware with SSD's.<br>
<br>
One of the theory's we have is that the inode tree of the filesystem no longer fits in memory which could result in lots of extra io's to the disks.<br>
Also, the object-replicator will walk through all the files effectively busting the inode-cache continuously.<br>
A way to test this theory is to add more memory to the nodes to see if this helps / moves the issue up a few million files but we haven't had the resources to test this out yet.<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
</div>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Paulo Ricardo</div><div><br></div>-- <br><span>European Master in Distributed Computing<i></i></span><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><br>
Royal Institute of Technology - <i style="font-style:normal">KTH</i><br></i></span><div><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><i style="font-style:normal">Instituto Superior Técnico - IST</i></i></span></div>
<div><span style="font-family:arial,sans-serif;line-height:15px"><i style="font-style:normal"><i style="font-style:normal"><a href="http://paulormg.com" target="_blank">http://paulormg.com</a></i></i></span></div><br>
</div>