[Openstack] [SWIFT] PUTs and GETs getting slower
Klaus Schürmann
klaus.schuermann at mediabeam.com
Tue Aug 6 05:36:45 UTC 2013
The proxys are in a normal condition:
top - 07:23:42 up 70 days, 15:43, 1 user, load average: 0.42, 0.38, 0.33
Tasks: 244 total, 2 running, 242 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 0.1 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 16335416 total, 3836272 used, 12499144 free, 179904 buffers
KiB Swap: 16678908 total, 0 used, 16678908 free, 2686964 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31255 swift 20 0 78932 17m 2080 S 4 0.1 105:30.16 swift-proxy-ser
31224 swift 20 0 78432 17m 2080 S 2 0.1 104:07.90 swift-proxy-ser
31221 swift 20 0 78056 16m 2080 S 2 0.1 118:23.36 swift-proxy-ser
31216 swift 20 0 78452 17m 2088 S 1 0.1 121:49.80 swift-proxy-ser
31258 swift 20 0 78560 17m 2088 S 1 0.1 115:10.25 swift-proxy-ser
31226 swift 20 0 78084 16m 2080 S 1 0.1 106:05.50 swift-proxy-ser
31227 swift 20 0 78824 17m 2092 S 1 0.1 107:07.37 swift-proxy-ser
I changed the container-server.conf and increased the replicators concurrency to 16.
But the background process still takes a lot of time and is running continuously:
Aug 5 21:59:05 storage-node1 container-server Beginning replication run
Aug 5 22:13:20 storage-node1 container-server Attempted to replicate 194410 dbs in 854.55144 seconds (227.49947/s)
Aug 5 22:13:20 storage-node1 container-server no_change:388814 ts_repl:0 diff:6 rsync:0 diff_capped:0 hashmatch:0 empty:0
Aug 5 22:13:20 storage-node1 container-server Beginning replication run
Aug 5 22:26:39 storage-node1 container-server Attempted to replicate 194420 dbs in 799.42965 seconds (243.19839/s)
Aug 5 22:26:39 storage-node1 container-server no_change:388831 ts_repl:0 diff:9 rsync:0 diff_capped:0 hashmatch:0 empty:0
@Morten: my 8.000.000 files are also split on 270.000 containers. But daily I get 1.500.000 new files and delete about 1.000.000 files.
Von: Leandro Reox [mailto:leandro.reox at gmail.com]
Gesendet: Dienstag, 6. August 2013 01:18
An: Morten Møller Riis
Cc: John Dickinson; Klaus Schürmann; Jonathan Lu; openstack at lists.openstack.org
Betreff: Re: [Openstack] [SWIFT] PUTs and GETs getting slower
+ Check the proxys ram ... if there starving, try restarting them and check if your performance recover
On Mon, Aug 5, 2013 at 7:10 PM, Morten Møller Riis <mmr at gigahost.dk> wrote:
I second this.
We have millions of objects on several accounts. Eg. one account has about 7 million images. These are split into containers according to their filename:
/images_000/something.jpg
...
/images_fff/something_else.jpg
Also, I recommend looking at tuning the configs. Some examples:
object-server.conf:
[object-replicator]
concurrency = 4
run_pause = 60
[object-updater]
concurrency = 4
[object-auditor]
files_per_second = 5
bytes_per_second = 2500000
concurrency = 25
container-server.conf:
[container-replicator]
concurrency = 8
run_pause = 60
[container-updater]
concurrency = 4
[container-sync]
interval = 300
container_time = 60
account-server.conf:
[account-replicator]
concurrency = 4
run_pause = 45
[account-reaper]
concurrency = 25
Mvh / Best regards
Morten Møller Riis
Gigahost ApS
mmr at gigahost.dk
On Aug 6, 2013, at 12:15 AM, John Dickinson <me at not.mn> wrote:
There are a few things you can look at.
First, the slowdowns you are seeing may be because of the increased numbers of files on disk. This causes the background processes to do more work, which can cause some issues for the server processes.
Second (and perhaps more likely), how is your data arranged? Are you putting every object into the same container? Swift is optimized for access across the entire namespace, so spreading your data across many containers can help improve write performance. Since you are running with account and proxy servers on the same spinning media as the object servers, as you get your cluster more full, write performance will degrade significantly with many objects in a single container. With your setup, you should probably have a max of about a million objects in a single container. If you separate your account and container servers to use SSDs (and perhaps separate machines), you could do 100x this number effectively. Note that this issue doesn't affect read performance. Object eead performance should stay high no matter the cardinality of objects in a single container.
--John
On Aug 5, 2013, at 4:09 AM, Jonathan Lu <jojokururu at gmail.com> wrote:
Hi,
I suffer the same issue that our swift system gets slower and slower. I tried almost all the guide in the document but it does little help.
Maybe swift is not suitable to the circumstance of putting and getting many little files. Maybe is it more suitable to the large object such as the image of OS? I am not quite sure.
Best Regards
Jonathan Lu
On 2013/8/5 18:35, Hua ZZ Zhang wrote:
Here's the deployment guide which can give you options for tuning container replicator and auditor, such as concurrency.
http://docs.openstack.org/developer/swift/deployment_guide.html
hope this can help you.
Best Regards,
Edward Zhang(张华)
Advisory Software Engineer
Software Standards & Open Source Software
Emerging Technology Institute(ETI)
IBM China Software Development Lab
e-mail: zhuadl at cn.ibm.com
<Mail Attachment.gif>
<Mail Attachment.gif>
<Mail Attachment.gif>
<Mail Attachment.gif>
<Mail Attachment.gif>Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus Schürmann <klaus.schuermann at mediabeam.com>
Klaus Schürmann <klaus.schuermann at mediabeam.com>
08/05/2013 06:14 PM
<Mail Attachment.gif>
To
<Mail Attachment.gif>
Hua ZZ Zhang/China/IBM at IBMCN,
<Mail Attachment.gif>
cc
<Mail Attachment.gif>
"openstack at lists.openstack.org" <openstack at lists.openstack.org>
<Mail Attachment.gif>
Subject
<Mail Attachment.gif>
AW: [Openstack] [SWIFT] PUTs and GETs getting slower
<Mail Attachment.gif> <Mail Attachment.gif>
Here is my container-server.conf:
[DEFAULT]
log_name = container-server
log_facility = LOG_LOCAL6
log_level = INFO
log_address = /dev/log
bind_ip = 10.12.1.1
workers = 8
log_statsd_host = 10.11.2.1
log_statsd_port = 8125
log_statsd_default_sample_rate = 0.25
log_statsd_metric_prefix = Storage01
[pipeline:main]
pipeline = recon container-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
[app:container-server]
use = egg:swift#container
[container-replicator]
concurrency = 2
[container-updater]
concurrency = 1
[container-auditor]
[container-sync]
--------
For the second question, Swift doesn't use updatedb.mlocate in my mind. it should be safe to disable it.
Based on the TOP output, it seems that the CPU workload of container replicator and auditor are too high.
Perhaps you need to tune the config file to decrease their workload. paste your config file of
container servers.
Best Regards,
Edward Zhang(张华)
Advisory Software Engineer
Software Standards & Open Source Software
Emerging Technology Institute(ETI)
IBM China Software Development Lab
e-mail: zhuadl at cn.ibm.com
<Mail Attachment.gif>
<Mail Attachment.gif>
<Mail Attachment.gif>
<Mail Attachment.gif> <Mail Attachment.gif>
<Mail Attachment.gif>Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus Schürmann <klaus.schuermann at mediabeam.com>
Klaus Schürmann <klaus.schuermann at mediabeam.com>
08/05/2013 05:26 PM
<Mail Attachment.gif>
To
<Mail Attachment.gif>
"openstack at lists.openstack.org" <openstack at lists.openstack.org>,
<Mail Attachment.gif>
cc
<Mail Attachment.gif>
<Mail Attachment.gif>
Subject
<Mail Attachment.gif>
[Openstack] [SWIFT] PUTs and GETs getting slower
<Mail Attachment.gif> <Mail Attachment.gif>
Hi,
my Swift cluster is working for three weeks now as a mail storage and the performance is getting slower and slower.
This is my hardware configuration:
- 1 hardware-loadbalancer
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA 6GB/s
The cluster contains:
279.000 containers
8.170.000 objects = 984 GByte
I have about 3.000.000 GETs per day and 1.500.000 PUTs.
The load is raising up to 12 now:
top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26, 11.22
Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers
KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf
24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf
29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid
29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1]
30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1]
6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0]
7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf
1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1]
9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0]
1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5
6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2]
9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0]
4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2]
5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3]
3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0]
52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1]
7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf
9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93 /usr/bin/updatedb.mlocate
Is this a memory problem or a harddisk problem?
And second question: On the server the mlocate database is rising. Can I disable the updatedb.mlocate process and empty the database?
Thanks
Klaus
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to :
openstack at lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
More information about the Openstack
mailing list