<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi, Huang<br>
Thanks for you explanation. Does it mean that the storage
cluster of specific processing ability will be slower and slower
with more and more objects? Is there any test about the rate of
the decline or is there any lower limit?<br>
<br>
For example, my environment is:<br>
<pre style="font-family: 'Ubuntu Mono', monospace; margin-bottom: 0.8em; color: rgb(51, 51, 51); font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 18px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Swift version : grizzly
Tried on Ubuntu 12.04
3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12
</pre>
The expected<b> </b>throughout is more than 100/s with
uploaded objects of 50KB. At the beginning it works quite well and
then it drops. If this degradation is unstoppable, I'm afraid that
the performance will finally not be able to meet our needs no
matter how I tuning other config.<br>
<br>
Jonathan Lu<br>
<br>
On 2013/6/18 11:05, Huang Zhiteng wrote:<br>
</div>
<blockquote
cite="mid:CAE7zfpMCR2Hce8fOfovT0UORuohx-U8TnfQ3YEMOHGo9Se0wZQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 18, 2013 at 10:42 AM,
Jonathan Lu <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jojokururu@gmail.com" target="_blank">jojokururu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 2013/6/17 18:59, Robert van Leeuwen
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm facing the issue about the performance
degradation, and once I glanced that changing the
value in /proc/sys<br>
/vm/vfs_cache_pressure will do a favour.<br>
Can anyone explain to me whether and why it is
useful?<br>
</blockquote>
Hi,<br>
<br>
When this is set to a lower value the kernel will try
to keep the inode/dentry cache longer in memory.<br>
Since the swift replicator is scanning the filesystem
continuously it will eat up a lot of iops if those are
not in memory.<br>
<br>
To see if a lot of cache misses are happening, for
xfs, you can look at xs_dir_lookup and xs_ig_missed.<br>
( look at <a moz-do-not-send="true"
href="http://xfs.org/index.php/Runtime_Stats"
target="_blank">http://xfs.org/index.php/Runtime_Stats</a>
)<br>
<br>
We greatly benefited from setting this to a low value
but we have quite a lot of files on a node ( 30
million)<br>
Note that setting this to zero will result in the OOM
killer killing the machine sooner or later.<br>
(especially if files are moved around due to a cluster
change ;)<br>
<br>
Cheers,<br>
Robert van Leeuwen<br>
</blockquote>
<br>
</div>
Hi,<br>
We set this to a low value(20) and the performance is
better than before. It seems quite useful.<br>
<br>
According to your description, this issue is related
with the object quantity in the storage. We delete all the
objects in the storage but it doesn't help anything. The
only method to recover is to format and re-mount the
storage node. We try to install swift on different
environment but this degradation problem seems to be an
inevitable one.<br>
</blockquote>
<div>It's inode cache for each file(object) helps (reduce
extra disk IOs). As long as your memory is big enough to
hold inode information of those frequently accessed
objects, you are good. And there's no need (no point) to
limit # of objects for each storage node IMO. You may
manually load inode information of objects into VFS cache
if you like (by simply 'ls' files), to _restore_
performance. But still memory size and object access
pattern are the key to this kind of performance tuning, if
memory is too small, inode cache will be invalided sooner
or later.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Cheers,<br>
Jonathan Lu
<div class="HOEnZb">
<div class="h5"><br>
<br>
_______________________________________________<br>
Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp"
target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
Regards<br>
Huang Zhiteng
</div>
</div>
</blockquote>
<br>
</body>
</html>