<html><body>
<p><font size="2" face="sans-serif">Here's the deployment guide which can give you options for tuning container replicator and auditor, such as </font><font size="1" face="Lucida Sans Unicode">concurrency</font><font size="3" face="serif">.</font><br>
<a href="http://docs.openstack.org/developer/swift/deployment_guide.html"><font size="3" color="#0000FF" face="serif"><u>http://docs.openstack.org/developer/swift/deployment_guide.html</u></font></a><font size="3" face="serif"> </font><br>
<font size="3" face="serif">hope this can help you.</font><br>
<font size="2" face="sans-serif"><br>
</font><font size="1" face="serif"><b>Best Regards, </b></font><font size="1" face="serif"><br>
<br>
</font><hr width="100%" size="2" align="left">
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="298" valign="middle">
<ul style="padding-left: 0pt"><font size="1" face="serif"><b>Edward Zhang(张华)</b></font><font size="1" face="serif"><br>
Advisory Software Engineer<br>
Software Standards & Open Source Software<br>
Emerging Technology Institute(ETI)<br>
IBM China Software Development Lab<br>
e-mail: zhuadl@cn.ibm.com<br>
</font></ul>
</td><td width="298" valign="middle"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
<tr valign="top"><td width="298" valign="middle">
<ul style="padding-left: 0pt"><img src="cid:2__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="100" height="100"></ul>
</td><td width="298" valign="middle"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
<tr valign="top"><td width="298" valign="middle">
<ul style="padding-left: 0pt"><font size="1" face="serif"><br>
</font></ul>
</td><td width="298" valign="middle"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
</table>
<br>
<img width="16" height="16" src="cid:3__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus Schürmann <klaus.schuermann@mediabeam.com>"><font size="2" color="#424282" face="sans-serif">Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus Schürmann <klaus.schuermann@mediabeam.com></font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:4__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com); background-repeat: no-repeat; " width="40%">
<ul style="padding-left: 72pt"><font size="1" face="sans-serif"><b>Klaus Schürmann <klaus.schuermann@mediabeam.com></b></font><font size="1" face="sans-serif"> </font>
<p><font size="1" face="sans-serif">08/05/2013 06:14 PM</font></ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">Hua ZZ Zhang/China/IBM@IBMCN, </font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">"openstack@lists.openstack.org" <openstack@lists.openstack.org></font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">AW: [Openstack] [SWIFT] PUTs and GETs getting slower</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<font size="2" color="#1F497D" face="Calibri">Here is my container-server.conf:</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[DEFAULT]</font><br>
<font size="2" color="#1F497D" face="Calibri">log_name = container-server</font><br>
<font size="2" color="#1F497D" face="Calibri">log_facility = LOG_LOCAL6</font><br>
<font size="2" color="#1F497D" face="Calibri">log_level = INFO</font><br>
<font size="2" color="#1F497D" face="Calibri">log_address = /dev/log</font><br>
<font size="2" color="#1F497D" face="Calibri">bind_ip = 10.12.1.1</font><br>
<font size="2" color="#1F497D" face="Calibri">workers = 8</font><br>
<font size="2" color="#1F497D" face="Calibri">log_statsd_host = 10.11.2.1</font><br>
<font size="2" color="#1F497D" face="Calibri">log_statsd_port = 8125</font><br>
<font size="2" color="#1F497D" face="Calibri">log_statsd_default_sample_rate = 0.25</font><br>
<font size="2" color="#1F497D" face="Calibri">log_statsd_metric_prefix = Storage01</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[pipeline:main]</font><br>
<font size="2" color="#1F497D" face="Calibri">pipeline = recon container-server</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[filter:recon]</font><br>
<font size="2" color="#1F497D" face="Calibri">use = egg:swift#recon</font><br>
<font size="2" color="#1F497D" face="Calibri">recon_cache_path = /var/cache/swift</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[app:container-server]</font><br>
<font size="2" color="#1F497D" face="Calibri">use = egg:swift#container</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[container-replicator]</font><br>
<font size="2" color="#1F497D" face="Calibri">concurrency = 2</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[container-updater]</font><br>
<font size="2" color="#1F497D" face="Calibri">concurrency = 1</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[container-auditor]</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">[container-sync]</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">--------</font>
<p><font size="2" face="Arial">For the second question, Swift doesn't use </font><font size="2" face="Courier New"> updatedb.mlocate</font><font size="2" face="Arial"> in my mind. it should be safe to disable it.<br>
Based on the TOP output, it seems that the CPU workload of container replicator and auditor are too high.<br>
Perhaps you need to tune the config file to decrease their workload. paste your config file of<br>
container servers.<br>
</font><font size="1" face="Times New Roman"><b><br>
Best Regards, </b></font><br>
<hr width="100%" size="2" align="left">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="298" valign="middle">
<ul style="padding-left: 36pt"><font size="1" face="Times New Roman"><b>Edward Zhang(</b></font><font size="1" face="PMingLiU"><b>张华</b></font><font size="1" face="Times New Roman"><b>)</b></font><font size="1" face="Times New Roman"><br>
Advisory Software Engineer<br>
Software Standards & Open Source Software<br>
Emerging Technology Institute(ETI)<br>
IBM China Software Development Lab<br>
e-mail: </font><a href="mailto:zhuadl@cn.ibm.com"><font size="1" color="#0000FF" face="Times New Roman"><u>zhuadl@cn.ibm.com</u></font></a></ul>
</td><td width="298" valign="middle"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td></tr>
<tr valign="top"><td width="298" valign="middle">
<ul style="padding-left: 36pt"><img src="cid:2__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="100" height="100"></ul>
</td><td width="298" valign="middle"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td></tr>
<tr valign="top"><td width="298" valign="middle"><img width="1" height="1" src="cid:1__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="298" valign="middle"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td></tr>
</table>
<font size="3" face="Times New Roman"><br>
</font><img src="cid:3__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="16" height="16" alt="Inactive hide details for Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus Schürmann <klaus.schuermann@mediabeam.com>"><font size="2" color="#424282" face="Arial">Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus Schürmann <</font><a href="mailto:klaus.schuermann@mediabeam.com"><font size="2" color="#0000FF" face="Arial"><u>klaus.schuermann@mediabeam.com</u></font></a><font size="2" color="#424282" face="Arial">></font>
<p>
<table class="MsoNormalTable" width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="46%">
<ul style="padding-left: 36pt"><font size="1" face="Arial"><b>Klaus Schürmann <</b></font><a href="mailto:klaus.schuermann@mediabeam.com"><font size="1" color="#0000FF" face="Arial"><b><u>klaus.schuermann@mediabeam.com</u></b></font></a><font size="1" face="Arial"><b>></b></font><font size="1" face="Arial"> </font><font size="3" face="Times New Roman"> </font>
<p><font size="1" face="Arial">08/05/2013 05:26 PM</font></ul>
</td><td width="54%">
<table class="MsoNormalTable" width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="13%"><img src="cid:6__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="Arial">To</font></div></td><td width="87%"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="Arial">"</font><a href="mailto:openstack@lists.openstack.org"><font size="1" color="#0000FF" face="Arial"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="Arial">" <</font><a href="mailto:openstack@lists.openstack.org"><font size="1" color="#0000FF" face="Arial"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="Arial">>, </font></ul>
</td></tr>
<tr valign="top"><td width="13%"><img src="cid:6__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="Arial">cc</font></div></td><td width="87%"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td></tr>
<tr valign="top"><td width="13%"><img src="cid:6__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="Arial">Subject</font></div></td><td width="87%"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="Arial">[Openstack] [SWIFT] PUTs and GETs getting slower</font></ul>
</td></tr>
</table>
<font size="3" face="Times New Roman"> </font>
<p><br>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td><td width="336"><img src="cid:5__=C7BBF12DDFABFB5E8f9e8a93df938@cn.ibm.com" width="1" height="1"></td></tr>
</table>
</td></tr>
</table>
<font size="2" face="Courier New"><br>
Hi,<br>
<br>
my Swift cluster is working for three weeks now as a mail storage and the performance is getting slower and slower.<br>
<br>
This is my hardware configuration:<br>
<br>
- 1 hardware-loadbalancer<br>
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU<br>
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA 6GB/s<br>
<br>
The cluster contains:<br>
<br>
279.000 containers<br>
8.170.000 objects = 984 GByte<br>
<br>
I have about 3.000.000 GETs per day and 1.500.000 PUTs.<br>
<br>
The load is raising up to 12 now:<br>
<br>
top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26, 11.22<br>
Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie<br>
%Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si, 0.0 st<br>
KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers<br>
KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf<br>
24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf<br>
29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid<br>
29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf<br>
30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1]<br>
30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf<br>
5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1]<br>
6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0]<br>
7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf<br>
1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1]<br>
9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf<br>
9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0]<br>
1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5<br>
6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2]<br>
9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf<br>
51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0]<br>
4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2]<br>
5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3]<br>
3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0]<br>
52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1]<br>
7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf<br>
9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf<br>
22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93 /usr/bin/updatedb.mlocate<br>
<br>
Is this a memory problem or a harddisk problem?<br>
<br>
And second question: On the server the mlocate database is rising. Can I disable the updatedb.mlocate process and empty the database?<br>
<br>
Thanks<br>
Klaus<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: </font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><font size="2" color="#0000FF" face="Courier New"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></a><font size="2" face="Courier New"><br>
Post to : </font><a href="mailto:openstack@lists.openstack.org"><font size="2" color="#0000FF" face="Courier New"><u>openstack@lists.openstack.org</u></font></a><font size="2" face="Courier New"><br>
Unsubscribe : </font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><font size="2" color="#0000FF" face="Courier New"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></a><font size="2" face="Courier New"><br>
</font><br>
</body></html>