<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi, Huang<br>
Thanks a lot. I will try this test.<br>
<br>
One more question:<br>
In the 3 following situation, will the base line performance
be quite different?<br>
1. only 1 sontaienr with 10m objects;<br>
2. 100,000 objects per container at 100 containers<br>
3. 1,000 objects per container at 10,000 containers<br>
<br>
Cheers<br>
Jonathan Lu<br>
<br>
On 2013/6/18 12:54, Huang Zhiteng wrote:<br>
</div>
<blockquote
cite="mid:CAE7zfpMyni7W8sQ3HuU48F9Pmb_cX1P9jWDQS4LwBkzDvixy7Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jun 18, 2013 at 12:35 PM,
Jonathan Lu <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi, Huang<br>
Thanks for you explanation. Does it mean that the
storage cluster of specific processing ability will be
slower and slower with more and more objects? Is there
any test about the rate of the decline or is there any
lower limit?<br>
<br>
For example, my environment is:<br>
<pre style="font-family:'Ubuntu Mono',monospace;margin-bottom:0.8em;color:rgb(51,51,51);font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:18px;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px"> Swift version : grizzly
Tried on Ubuntu 12.04
3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12
</pre>
The expected<b> </b>throughout is more than 100/s
with uploaded objects of 50KB. At the beginning it
works quite well and then it drops. If this
degradation is unstoppable, I'm afraid that the
performance will finally not be able to meet our needs
no matter how I tuning other config.<span
class="HOEnZb"><font color="#888888"><br>
<br>
</font></span></div>
</div>
</blockquote>
<div>It won't be hard to do a base line performance (without
inode cache) assessment of your system: populate your
system with certain mount of objects with desired size
(say 50k, 10million objects <1,000 objects per
container at 10,000 containers>), and *then drop VFS
caches explicitly before testing*. Measure performance
with your desired IO pattern and in the mean time drop VFS
cache every once in a while (say every 60s). That's
roughly the performance you can get when your storage
system gets into a 'steady' state (i.e. objects # has out
grown memory size). This will give you idea of pretty
much the worst case.<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><span class="HOEnZb"><font color="#888888">
Jonathan Lu</font></span>
<div>
<div class="h5"><br>
<br>
On 2013/6/18 11:05, Huang Zhiteng wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 18, 2013
at 10:42 AM, Jonathan Lu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:jojokururu@gmail.com"
target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div>On 2013/6/17 18:59, Robert van
Leeuwen wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> I'm facing
the issue about the performance
degradation, and once I glanced that
changing the value in /proc/sys<br>
/vm/vfs_cache_pressure will do a
favour.<br>
Can anyone explain to me whether and
why it is useful?<br>
</blockquote>
Hi,<br>
<br>
When this is set to a lower value the
kernel will try to keep the
inode/dentry cache longer in memory.<br>
Since the swift replicator is scanning
the filesystem continuously it will
eat up a lot of iops if those are not
in memory.<br>
<br>
To see if a lot of cache misses are
happening, for xfs, you can look at
xs_dir_lookup and xs_ig_missed.<br>
( look at <a moz-do-not-send="true"
href="http://xfs.org/index.php/Runtime_Stats"
target="_blank">http://xfs.org/index.php/Runtime_Stats</a>
)<br>
<br>
We greatly benefited from setting this
to a low value but we have quite a lot
of files on a node ( 30 million)<br>
Note that setting this to zero will
result in the OOM killer killing the
machine sooner or later.<br>
(especially if files are moved around
due to a cluster change ;)<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
</blockquote>
<br>
</div>
Hi,<br>
We set this to a low value(20) and the
performance is better than before. It
seems quite useful.<br>
<br>
According to your description, this
issue is related with the object quantity
in the storage. We delete all the objects
in the storage but it doesn't help
anything. The only method to recover is to
format and re-mount the storage node. We
try to install swift on different
environment but this degradation problem
seems to be an inevitable one.<br>
</blockquote>
<div>It's inode cache for each file(object)
helps (reduce extra disk IOs). As long as
your memory is big enough to hold inode
information of those frequently accessed
objects, you are good. And there's no
need (no point) to limit # of objects for
each storage node IMO. You may manually
load inode information of objects into VFS
cache if you like (by simply 'ls' files),
to _restore_ performance. But still
memory size and object access pattern are
the key to this kind of performance
tuning, if memory is too small, inode
cache will be invalided sooner or later.<br>
<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> Cheers,<br>
Jonathan Lu
<div>
<div><br>
<br>
_______________________________________________<br>
Mailing list: <a
moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a
moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a
moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp"
target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng
</div>
</div>
</blockquote>
<br>
</body>
</html>