<div dir="ltr">Jonathan, we happen to use SN similar with yours and I could share you some performance testing data here:<br><div><div class="gmail_extra"><br></div><div class="gmail_extra">1. 100 container with 10000 objects(base)<br>
</div><div class="gmail_extra">The performance is quite good and can hit HW bottleneck<br><br></div><div class="gmail_extra">2. 10kcontainer with 100M objects<br></div><div class="gmail_extra">The performance is not so good, which dropped 80% compared with base <br>
<br></div><div class="gmail_extra">3. 1container with 1000M objects<br></div><div class="gmail_extra">The performance is not so good, which dropped 95% compared with base<br></div><div class="gmail_extra"><br>The suspect reason we found are:<br>
1) XFS's overhead W/ huge # of objects. Deleting some files wouldn't help since the inode allocation is quite sparse on the disks already and later inode_lookup should costs more disk seek time I guess. But this could be greatly improved by setting vfs_cache_presure to a lower value and it should be safe even we set it to 1 since Swift does use cache at all. If we could cache all the inode and the performance would become good again.We've done some tests with precached inode(simply run 'ls -R /srv/nodes') and verified the performance is quite good.<br>
<br></div><div class="gmail_extra">2) SQLite DB performance bottleneck when there are millions of records in a single DB. There is a BP to auto split large database but not implemented yet<br></div><div class="gmail_extra">
<br></div><div class="gmail_extra"><br>Hope this can help.<br></div><div class="gmail_extra"><br>-- <br>Sincerely, Yuan <br><br><div class="gmail_quote">On Tue, Jun 18, 2013 at 1:56 PM, Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Huang<br>
Thanks a lot. I will try this test.<br>
<br>
One more question:<br>
In the 3 following situation, will the base line performance
be quite different?<br>
1. only 1 sontaienr with 10m objects;<br>
2. 100,000 objects per container at 100 containers<br>
3. 1,000 objects per container at 10,000 containers<br>
<br>
Cheers<span class=""><font color="#888888"><br>
Jonathan Lu</font></span><div><div class="h5"><br>
<br>
On 2013/6/18 12:54, Huang Zhiteng wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jun 18, 2013 at 12:35 PM,
Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Huang<br>
Thanks for you explanation. Does it mean that the
storage cluster of specific processing ability will be
slower and slower with more and more objects? Is there
any test about the rate of the decline or is there any
lower limit?<br>
<br>
For example, my environment is:<br>
<pre style="font-family:'Ubuntu Mono',monospace;margin-bottom:0.8em;color:rgb(51,51,51);font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:18px;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px">
Swift version : grizzly
Tried on Ubuntu 12.04
3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12
</pre>
The expected<b> </b>throughout is more than 100/s
with uploaded objects of 50KB. At the beginning it
works quite well and then it drops. If this
degradation is unstoppable, I'm afraid that the
performance will finally not be able to meet our needs
no matter how I tuning other config.<span><font color="#888888"><br>
<br>
</font></span></div>
</div>
</blockquote>
<div>It won't be hard to do a base line performance (without
inode cache) assessment of your system: populate your
system with certain mount of objects with desired size
(say 50k, 10million objects <1,000 objects per
container at 10,000 containers>), and *then drop VFS
caches explicitly before testing*. Measure performance
with your desired IO pattern and in the mean time drop VFS
cache every once in a while (say every 60s). That's
roughly the performance you can get when your storage
system gets into a 'steady' state (i.e. objects # has out
grown memory size). This will give you idea of pretty
much the worst case.<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><span><font color="#888888">
Jonathan Lu</font></span>
<div>
<div><br>
<br>
On 2013/6/18 11:05, Huang Zhiteng wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 18, 2013
at 10:42 AM, Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>On 2013/6/17 18:59, Robert van
Leeuwen wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> I'm facing
the issue about the performance
degradation, and once I glanced that
changing the value in /proc/sys<br>
/vm/vfs_cache_pressure will do a
favour.<br>
Can anyone explain to me whether and
why it is useful?<br>
</blockquote>
Hi,<br>
<br>
When this is set to a lower value the
kernel will try to keep the
inode/dentry cache longer in memory.<br>
Since the swift replicator is scanning
the filesystem continuously it will
eat up a lot of iops if those are not
in memory.<br>
<br>
To see if a lot of cache misses are
happening, for xfs, you can look at
xs_dir_lookup and xs_ig_missed.<br>
( look at <a href="http://xfs.org/index.php/Runtime_Stats" target="_blank">http://xfs.org/index.php/Runtime_Stats</a>
)<br>
<br>
We greatly benefited from setting this
to a low value but we have quite a lot
of files on a node ( 30 million)<br>
Note that setting this to zero will
result in the OOM killer killing the
machine sooner or later.<br>
(especially if files are moved around
due to a cluster change ;)<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
</blockquote>
<br>
</div>
Hi,<br>
We set this to a low value(20) and the
performance is better than before. It
seems quite useful.<br>
<br>
According to your description, this
issue is related with the object quantity
in the storage. We delete all the objects
in the storage but it doesn't help
anything. The only method to recover is to
format and re-mount the storage node. We
try to install swift on different
environment but this degradation problem
seems to be an inevitable one.<br>
</blockquote>
<div>It's inode cache for each file(object)
helps (reduce extra disk IOs). As long as
your memory is big enough to hold inode
information of those frequently accessed
objects, you are good. And there's no
need (no point) to limit # of objects for
each storage node IMO. You may manually
load inode information of objects into VFS
cache if you like (by simply 'ls' files),
to _restore_ performance. But still
memory size and object access pattern are
the key to this kind of performance
tuning, if memory is too small, inode
cache will be invalided sooner or later.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Cheers,<br>
Jonathan Lu
<div>
<div><br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng
</div>
</div>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br><br clear="all"><br><br>
</div></div></div>