[Openstack] [SWIFT] PUTs and GETs getting slower
Leandro Reox
leandro.reox at gmail.com
Mon Aug 5 23:17:57 UTC 2013
+ Check the proxys ram ... if there starving, try restarting them and check
if your performance recover
On Mon, Aug 5, 2013 at 7:10 PM, Morten Møller Riis <mmr at gigahost.dk> wrote:
> I second this.
>
> We have millions of objects on several accounts. Eg. one account has about
> 7 million images. These are split into containers according to their
> filename:
>
> /images_000/something.jpg
> ...
> /images_fff/something_else.jpg
>
> Also, I recommend looking at tuning the configs. Some examples:
>
>
> object-server.conf:
>
> [object-replicator]
> concurrency = 4
> run_pause = 60
>
> [object-updater]
> concurrency = 4
>
> [object-auditor]
> files_per_second = 5
> bytes_per_second = 2500000
> concurrency = 25
>
>
> container-server.conf:
>
> [container-replicator]
> concurrency = 8
> run_pause = 60
>
> [container-updater]
> concurrency = 4
>
> [container-sync]
> interval = 300
> container_time = 60
>
>
> account-server.conf:
>
> [account-replicator]
> concurrency = 4
> run_pause = 45
>
> [account-reaper]
> concurrency = 25
>
>
>
> Mvh / Best regards
> Morten Møller Riis
> Gigahost ApS
> mmr at gigahost.dk
>
>
>
>
> On Aug 6, 2013, at 12:15 AM, John Dickinson <me at not.mn> wrote:
>
> There are a few things you can look at.
>
> First, the slowdowns you are seeing may be because of the increased
> numbers of files on disk. This causes the background processes to do more
> work, which can cause some issues for the server processes.
>
> Second (and perhaps more likely), how is your data arranged? Are you
> putting every object into the same container? Swift is optimized for access
> across the entire namespace, so spreading your data across many containers
> can help improve write performance. Since you are running with account and
> proxy servers on the same spinning media as the object servers, as you get
> your cluster more full, write performance will degrade significantly with
> many objects in a single container. With your setup, you should probably
> have a max of about a million objects in a single container. If you
> separate your account and container servers to use SSDs (and perhaps
> separate machines), you could do 100x this number effectively. Note that
> this issue doesn't affect read performance. Object eead performance should
> stay high no matter the cardinality of objects in a single container.
>
>
> --John
>
>
>
> On Aug 5, 2013, at 4:09 AM, Jonathan Lu <jojokururu at gmail.com> wrote:
>
> Hi,
> I suffer the same issue that our swift system gets slower and slower. I
> tried almost all the guide in the document but it does little help.
>
> Maybe swift is not suitable to the circumstance of putting and getting
> many little files. Maybe is it more suitable to the large object such as
> the image of OS? I am not quite sure.
>
> Best Regards
> Jonathan Lu
>
> On 2013/8/5 18:35, Hua ZZ Zhang wrote:
>
> Here's the deployment guide which can give you options for tuning
> container replicator and auditor, such as concurrency.
> http://docs.openstack.org/developer/swift/deployment_guide.html
> hope this can help you.
>
> Best Regards,
>
>
> Edward Zhang(张华)
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: zhuadl at cn.ibm.com
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif>
>
> <Mail Attachment.gif>
> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus
> Schürmann <klaus.schuermann at mediabeam.com>
>
> Klaus Schürmann <klaus.schuermann at mediabeam.com>
> 08/05/2013 06:14 PM
>
> <Mail Attachment.gif>
> To
> <Mail Attachment.gif>
> Hua ZZ Zhang/China/IBM at IBMCN,
> <Mail Attachment.gif>
> cc
> <Mail Attachment.gif>
> "openstack at lists.openstack.org" <openstack at lists.openstack.org>
> <Mail Attachment.gif>
> Subject
> <Mail Attachment.gif>
> AW: [Openstack] [SWIFT] PUTs and GETs getting slower
> <Mail Attachment.gif> <Mail Attachment.gif>
> Here is my container-server.conf:
>
> [DEFAULT]
> log_name = container-server
> log_facility = LOG_LOCAL6
> log_level = INFO
> log_address = /dev/log
> bind_ip = 10.12.1.1
> workers = 8
> log_statsd_host = 10.11.2.1
> log_statsd_port = 8125
> log_statsd_default_sample_rate = 0.25
> log_statsd_metric_prefix = Storage01
>
> [pipeline:main]
> pipeline = recon container-server
>
> [filter:recon]
> use = egg:swift#recon
> recon_cache_path = /var/cache/swift
>
> [app:container-server]
> use = egg:swift#container
>
> [container-replicator]
> concurrency = 2
>
> [container-updater]
> concurrency = 1
>
> [container-auditor]
>
> [container-sync]
>
>
> --------
> For the second question, Swift doesn't use updatedb.mlocate in my mind.
> it should be safe to disable it.
> Based on the TOP output, it seems that the CPU workload of container
> replicator and auditor are too high.
> Perhaps you need to tune the config file to decrease their workload. paste
> your config file of
> container servers.
>
> Best Regards,
> Edward Zhang(张华)
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: zhuadl at cn.ibm.com
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif> <Mail Attachment.gif>
> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus
> Schürmann <klaus.schuermann at mediabeam.com>
> Klaus Schürmann <klaus.schuermann at mediabeam.com>
> 08/05/2013 05:26 PM
>
> <Mail Attachment.gif>
> To
> <Mail Attachment.gif>
> "openstack at lists.openstack.org" <openstack at lists.openstack.org>,
> <Mail Attachment.gif>
> cc
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> Subject
> <Mail Attachment.gif>
> [Openstack] [SWIFT] PUTs and GETs getting slower
>
>
> <Mail Attachment.gif> <Mail Attachment.gif>
> Hi,
>
> my Swift cluster is working for three weeks now as a mail storage and the
> performance is getting slower and slower.
>
> This is my hardware configuration:
>
> - 1 hardware-loadbalancer
> - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
> - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA
> 6GB/s
>
> The cluster contains:
>
> 279.000 containers
> 8.170.000 objects = 984 GByte
>
> I have about 3.000.000 GETs per day and 1.500.000 PUTs.
>
> The load is raising up to 12 now:
>
> top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26,
> 11.22
> Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si,
> 0.0 st
> KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers
> KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36
> /usr/bin/python /usr/bin/swift-container-replicator
> /etc/swift/container-server.conf
> 24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23
> /usr/bin/python /usr/bin/swift-container-auditor
> /etc/swift/container-server.conf
> 29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90
> /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p
> /var/run/snmpd.pid
> 29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1]
> 30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1]
> 6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0]
> 7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1]
> 9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16
> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
> 9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0]
> 1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5
> 6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2]
> 9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0]
> 4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2]
> 5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3]
> 3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0]
> 52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1]
> 7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06
> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
> 22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93
> /usr/bin/updatedb.mlocate
>
> Is this a memory problem or a harddisk problem?
>
> And second question: On the server the mlocate database is rising. Can I
> disable the updatedb.mlocate process and empty the database?
>
> Thanks
> Klaus
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
> Post to :
> openstack at lists.openstack.org
>
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/54776f25/attachment.html>
More information about the Openstack
mailing list