<div dir="ltr"><div>Jonathan, <br><br></div>Yes we have 10,000 containers and we're using COSBench to do the tests.<br><div class="gmail_extra"><br clear="all"><div><br>Sincerely, Yuan <br></div>
<br><br><div class="gmail_quote">On Wed, Jun 19, 2013 at 9:24 AM, Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Zhou,<br>
BTW, in test case 2, the number of container is 10,000 or just
10?<div class="im"><br>
<br>
Jonathan Lu<br>
<br>
On 2013/6/18 19:18, ZHOU Yuan wrote:<br>
</div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">Jonathan, we happen to use SN similar with yours
and I could share you some performance testing data here:<br>
<div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">1. 100 container with 10000
objects(base)<br>
</div>
<div class="gmail_extra">The performance is quite good and can
hit HW bottleneck<br>
<br>
</div>
<div class="gmail_extra">2. 10kcontainer with 100M objects<br>
</div>
<div class="gmail_extra">The performance is not so good, which
dropped 80% compared with base <br>
<br>
</div>
<div class="gmail_extra">3. 1container with 1000M objects<br>
</div>
<div class="gmail_extra">The performance is not so good, which
dropped 95% compared with base<br>
</div>
<div class="gmail_extra"><br>
The suspect reason we found are:<br>
1) XFS's overhead W/ huge # of objects. Deleting some files
wouldn't help since the inode allocation is quite sparse on
the disks already and later inode_lookup should costs more
disk seek time I guess. But this could be greatly improved
by setting vfs_cache_presure to a lower value and it should
be safe even we set it to 1 since Swift does use cache at
all. If we could cache all the inode and the performance
would become good again.We've done some tests with precached
inode(simply run 'ls -R /srv/nodes') and verified the
performance is quite good.<br>
<br>
</div>
<div class="gmail_extra">2) SQLite DB performance bottleneck
when there are millions of records in a single DB. There is
a BP to auto split large database but not implemented yet<br>
</div>
<div class="gmail_extra">
<br>
</div>
<div class="gmail_extra"><br>
Hope this can help.<br>
</div>
<div class="gmail_extra"><br>
-- <br>
Sincerely, Yuan <br>
<br>
<div class="gmail_quote">On Tue, Jun 18, 2013 at 1:56 PM,
Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Huang<br>
Thanks a lot. I will try this test.<br>
<br>
One more question:<br>
In the 3 following situation, will the base line
performance be quite different?<br>
1. only 1 sontaienr with 10m objects;<br>
2. 100,000 objects per container at 100
containers<br>
3. 1,000 objects per container at 10,000
containers<br>
<br>
Cheers<span><font color="#888888"><br>
Jonathan Lu</font></span>
<div>
<div><br>
<br>
On 2013/6/18 12:54, Huang Zhiteng wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jun 18,
2013 at 12:35 PM, Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Huang<br>
Thanks for you explanation. Does
it mean that the storage cluster of
specific processing ability will be
slower and slower with more and more
objects? Is there any test about the
rate of the decline or is there any
lower limit?<br>
<br>
For example, my environment is:<br>
<pre style="font-family:'Ubuntu Mono',monospace;margin-bottom:0.8em;color:rgb(51,51,51);font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:18px;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px">
Swift version : grizzly
Tried on Ubuntu 12.04
3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12
</pre>
The expected<b> </b>throughout
is more than 100/s with uploaded
objects of 50KB. At the beginning it
works quite well and then it drops.
If this degradation is unstoppable,
I'm afraid that the performance will
finally not be able to meet our
needs no matter how I tuning other
config.<span><font color="#888888"><br>
<br>
</font></span></div>
</div>
</blockquote>
<div>It won't be hard to do a base line
performance (without inode cache)
assessment of your system: populate your
system with certain mount of objects
with desired size (say 50k, 10million
objects <1,000 objects per container
at 10,000 containers>), and *then
drop VFS caches explicitly before
testing*. Measure performance with your
desired IO pattern and in the mean time
drop VFS cache every once in a while
(say every 60s). That's roughly the
performance you can get when your
storage system gets into a 'steady'
state (i.e. objects # has out grown
memory size). This will give you idea
of pretty much the worst case.<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><span><font color="#888888">
Jonathan Lu</font></span>
<div>
<div><br>
<br>
On 2013/6/18 11:05, Huang
Zhiteng wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Jun 18, 2013 at 10:42
AM, Jonathan Lu <span dir="ltr"><<a href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>On 2013/6/17 18:59,
Robert van Leeuwen
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I'm facing the
issue about the
performance
degradation, and
once I glanced
that changing the
value in /proc/sys<br>
/vm/vfs_cache_pressure
will do a favour.<br>
Can anyone explain
to me whether and
why it is useful?<br>
</blockquote>
Hi,<br>
<br>
When this is set to
a lower value the
kernel will try to
keep the
inode/dentry cache
longer in memory.<br>
Since the swift
replicator is
scanning the
filesystem
continuously it will
eat up a lot of iops
if those are not in
memory.<br>
<br>
To see if a lot of
cache misses are
happening, for xfs,
you can look at
xs_dir_lookup and
xs_ig_missed.<br>
( look at <a href="http://xfs.org/index.php/Runtime_Stats" target="_blank">http://xfs.org/index.php/Runtime_Stats</a>
)<br>
<br>
We greatly benefited
from setting this to
a low value but we
have quite a lot of
files on a node ( 30
million)<br>
Note that setting
this to zero will
result in the OOM
killer killing the
machine sooner or
later.<br>
(especially if files
are moved around due
to a cluster change
;)<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
</blockquote>
<br>
</div>
Hi,<br>
We set this to a low
value(20) and the
performance is better
than before. It seems
quite useful.<br>
<br>
According to your
description, this issue
is related with the
object quantity in the
storage. We delete all
the objects in the
storage but it doesn't
help anything. The only
method to recover is to
format and re-mount the
storage node. We try to
install swift on
different environment
but this degradation
problem seems to be an
inevitable one.<br>
</blockquote>
<div>It's inode cache for
each file(object) helps
(reduce extra disk
IOs). As long as your
memory is big enough to
hold inode information
of those frequently
accessed objects, you
are good. And there's
no need (no point) to
limit # of objects for
each storage node IMO.
You may manually load
inode information of
objects into VFS cache
if you like (by simply
'ls' files), to
_restore_ performance.
But still memory size
and object access
pattern are the key to
this kind of performance
tuning, if memory is too
small, inode cache will
be invalided sooner or
later.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Cheers,<br>
Jonathan Lu
<div>
<div><br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div>