[Openstack] [SWIFT] PUTs and GETs getting slower
Jonathan Lu
jojokururu at gmail.com
Mon Aug 5 11:09:49 UTC 2013
Hi,
I suffer the same issue that our swift system gets slower and
slower. I tried almost all the guide in the document but it does little
help.
Maybe swift is not suitable to the circumstance of putting and
getting many little files. Maybe is it more suitable to the large object
such as the image of OS? I am not quite sure.
Best Regards
Jonathan Lu
On 2013/8/5 18:35, Hua ZZ Zhang wrote:
>
> Here's the deployment guide which can give you options for tuning
> container replicator and auditor, such as concurrency.
> _http://docs.openstack.org/developer/swift/deployment_guide.html_
> hope this can help you.
>
> *Best Regards, *
>
> ------------------------------------------------------------------------
>
> *Edward Zhang(??)*
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: zhuadl at cn.ibm.com
>
>
>
>
>
>
>
>
>
> Inactive hide details for Klaus Schürmann ---08/05/2013 06:13:43
> PM---Klaus Schürmann <klaus.schuermann at mediabeam.com>Klaus Schürmann
> ---08/05/2013 06:13:43 PM---Klaus Schürmann
> <klaus.schuermann at mediabeam.com>
>
> *Klaus Schürmann <klaus.schuermann at mediabeam.com>*
>
> 08/05/2013 06:14 PM
>
>
>
> To
>
>
> Hua ZZ Zhang/China/IBM at IBMCN,
>
>
> cc
>
>
> "openstack at lists.openstack.org" <openstack at lists.openstack.org>
>
>
> Subject
>
>
> AW: [Openstack] [SWIFT] PUTs and GETs getting slower
>
>
>
>
> Here is my container-server.conf:
>
> [DEFAULT]
> log_name = container-server
> log_facility = LOG_LOCAL6
> log_level = INFO
> log_address = /dev/log
> bind_ip = 10.12.1.1
> workers = 8
> log_statsd_host = 10.11.2.1
> log_statsd_port = 8125
> log_statsd_default_sample_rate = 0.25
> log_statsd_metric_prefix = Storage01
>
> [pipeline:main]
> pipeline = recon container-server
>
> [filter:recon]
> use = egg:swift#recon
> recon_cache_path = /var/cache/swift
>
> [app:container-server]
> use = egg:swift#container
>
> [container-replicator]
> concurrency = 2
>
> [container-updater]
> concurrency = 1
>
> [container-auditor]
>
> [container-sync]
>
>
> --------
>
> For the second question, Swift doesn't use updatedb.mlocate in my
> mind. it should be safe to disable it.
> Based on the TOP output, it seems that the CPU workload of container
> replicator and auditor are too high.
> Perhaps you need to tune the config file to decrease their workload.
> paste your config file of
> container servers.
> *
> Best Regards, *
>
> ------------------------------------------------------------------------
>
> *Edward Zhang(**??**)*
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: _zhuadl at cn.ibm.com_ <mailto:zhuadl at cn.ibm.com>
>
>
>
>
>
>
>
> Inactive hide details for Klaus Schürmann ---08/05/2013 05:43:08
> PM---Klaus Schürmann <klaus.schuermann at mediabeam.com>Klaus Schürmann
> ---08/05/2013 05:43:08 PM---Klaus Schürmann
> <_klaus.schuermann at mediabeam.com_
> <mailto:klaus.schuermann at mediabeam.com>>
>
> *Klaus Schürmann <**_klaus.schuermann at mediabeam.com_*
> <mailto:klaus.schuermann at mediabeam.com>*>*
>
> 08/05/2013 05:26 PM
>
>
> To
>
>
> "_openstack at lists.openstack.org_
> <mailto:openstack at lists.openstack.org>"
> <_openstack at lists.openstack.org_
> <mailto:openstack at lists.openstack.org>>,
>
> cc
>
> Subject
>
>
> [Openstack] [SWIFT] PUTs and GETs getting slower
>
>
>
>
>
> Hi,
>
> my Swift cluster is working for three weeks now as a mail storage and
> the performance is getting slower and slower.
>
> This is my hardware configuration:
>
> - 1 hardware-loadbalancer
> - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
> - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD
> SATA 6GB/s
>
> The cluster contains:
>
> 279.000 containers
> 8.170.000 objects = 984 GByte
>
> I have about 3.000.000 GETs per day and 1.500.000 PUTs.
>
> The load is raising up to 12 now:
>
> top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88,
> 11.26, 11.22
> Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8
> si, 0.0 st
> KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers
> KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36
> /usr/bin/python /usr/bin/swift-container-replicator
> /etc/swift/container-server.conf
> 24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23
> /usr/bin/python /usr/bin/swift-container-auditor
> /etc/swift/container-server.conf
> 29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90
> /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p
> /var/run/snmpd.pid
> 29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1]
> 30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1]
> 6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0]
> 7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1]
> 9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16
> /usr/bin/python /usr/bin/swift-object-auditor
> /etc/swift/object-server.conf
> 9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13
> [kworker/3:0]
> 1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5
> 6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2]
> 9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08
> /usr/bin/python /usr/bin/swift-account-server
> /etc/swift/account-server.conf
> 51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0]
> 4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2]
> 5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3]
> 3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0]
> 52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1]
> 7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06
> /usr/bin/python /usr/bin/swift-object-auditor
> /etc/swift/object-server.conf
> 22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93
> /usr/bin/updatedb.mlocate
>
> Is this a memory problem or a harddisk problem?
>
> And second question: On the server the mlocate database is rising.
> Can I disable the updatedb.mlocate process and empty the database?
>
> Thanks
> Klaus
>
>
>
> _______________________________________________
> Mailing list:
> _http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack_
> Post to : _openstack at lists.openstack.org_
> <mailto:openstack at lists.openstack.org>
> Unsubscribe :
> _http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack_
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1279 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 166 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment-0003.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 168 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/97bcffc0/attachment-0004.gif>
More information about the Openstack
mailing list
