[Openstack] [SWIFT] PUTs and GETs getting slower

Leandro Reox leandro.reox at gmail.com
Mon Aug 5 23:17:57 UTC 2013


+ Check the proxys ram ... if there starving, try restarting them and check
if your performance recover


On Mon, Aug 5, 2013 at 7:10 PM, Morten Møller Riis <mmr at gigahost.dk> wrote:

> I second this.
>
> We have millions of objects on several accounts. Eg. one account has about
> 7 million images. These are split into containers according to their
> filename:
>
> /images_000/something.jpg
> ...
> /images_fff/something_else.jpg
>
> Also, I recommend looking at tuning the configs.  Some examples:
>
>
> object-server.conf:
>
> [object-replicator]
> concurrency = 4
> run_pause = 60
>
> [object-updater]
> concurrency = 4
>
> [object-auditor]
> files_per_second = 5
> bytes_per_second = 2500000
> concurrency = 25
>
>
> container-server.conf:
>
> [container-replicator]
> concurrency = 8
> run_pause = 60
>
> [container-updater]
> concurrency = 4
>
> [container-sync]
> interval = 300
> container_time = 60
>
>
> account-server.conf:
>
> [account-replicator]
> concurrency = 4
> run_pause = 45
>
> [account-reaper]
> concurrency = 25
>
>
>
> Mvh / Best regards
> Morten Møller Riis
> Gigahost ApS
> mmr at gigahost.dk
>
>
>
>
> On Aug 6, 2013, at 12:15 AM, John Dickinson <me at not.mn> wrote:
>
> There are a few things you can look at.
>
> First, the slowdowns you are seeing may be because of the increased
> numbers of files on disk. This causes the background processes to do more
> work, which can cause some issues for the server processes.
>
> Second (and perhaps more likely), how is your data arranged? Are you
> putting every object into the same container? Swift is optimized for access
> across the entire namespace, so spreading your data across many containers
> can help improve write performance. Since you are running with account and
> proxy servers on the same spinning media as the object servers, as you get
> your cluster more full, write performance will degrade significantly with
> many objects in a single container. With your setup, you should probably
> have a max of about a million objects in a single container. If you
> separate your account and container servers to use SSDs (and perhaps
> separate machines), you could do 100x this number effectively. Note that
> this issue doesn't affect read performance. Object eead performance should
> stay high no matter the cardinality of objects in a single container.
>
>
> --John
>
>
>
> On Aug 5, 2013, at 4:09 AM, Jonathan Lu <jojokururu at gmail.com> wrote:
>
> Hi,
>    I suffer the same issue that our swift system gets slower and slower. I
> tried almost all the guide in the document but it does little help.
>
>    Maybe swift is not suitable to the circumstance of putting and getting
> many little files. Maybe is it more suitable to the large object such as
> the image of OS? I am not quite sure.
>
> Best Regards
> Jonathan Lu
>
> On 2013/8/5 18:35, Hua ZZ Zhang wrote:
>
> Here's the deployment guide which can give you options for tuning
> container replicator and auditor, such as concurrency.
> http://docs.openstack.org/developer/swift/deployment_guide.html
> hope this can help you.
>
> Best Regards,
>
>
> Edward Zhang(张华)
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: zhuadl at cn.ibm.com
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif>
>
> <Mail Attachment.gif>
> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus
> Schürmann <klaus.schuermann at mediabeam.com>
>
> Klaus Schürmann <klaus.schuermann at mediabeam.com>
> 08/05/2013 06:14 PM
>
> <Mail Attachment.gif>
> To
> <Mail Attachment.gif>
> Hua ZZ Zhang/China/IBM at IBMCN,
> <Mail Attachment.gif>
> cc
> <Mail Attachment.gif>
> "openstack at lists.openstack.org" <openstack at lists.openstack.org>
> <Mail Attachment.gif>
> Subject
> <Mail Attachment.gif>
> AW: [Openstack] [SWIFT] PUTs and GETs getting slower
> <Mail Attachment.gif> <Mail Attachment.gif>
> Here is my container-server.conf:
>
> [DEFAULT]
> log_name = container-server
> log_facility = LOG_LOCAL6
> log_level = INFO
> log_address = /dev/log
> bind_ip = 10.12.1.1
> workers = 8
> log_statsd_host = 10.11.2.1
> log_statsd_port = 8125
> log_statsd_default_sample_rate = 0.25
> log_statsd_metric_prefix = Storage01
>
> [pipeline:main]
> pipeline = recon container-server
>
> [filter:recon]
> use = egg:swift#recon
> recon_cache_path = /var/cache/swift
>
> [app:container-server]
> use = egg:swift#container
>
> [container-replicator]
> concurrency = 2
>
> [container-updater]
> concurrency = 1
>
> [container-auditor]
>
> [container-sync]
>
>
> --------
> For the second question, Swift doesn't use  updatedb.mlocate in my mind.
> it should be safe to disable it.
> Based on the TOP output, it seems that the CPU workload of container
> replicator and auditor are too high.
> Perhaps you need to tune the config file to decrease their workload. paste
> your config file of
> container servers.
>
> Best Regards,
> Edward Zhang(张华)
> Advisory Software Engineer
> Software Standards & Open Source Software
> Emerging Technology Institute(ETI)
> IBM China Software Development Lab
> e-mail: zhuadl at cn.ibm.com
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> <Mail Attachment.gif> <Mail Attachment.gif>
> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus
> Schürmann <klaus.schuermann at mediabeam.com>
> Klaus Schürmann <klaus.schuermann at mediabeam.com>
> 08/05/2013 05:26 PM
>
> <Mail Attachment.gif>
> To
> <Mail Attachment.gif>
> "openstack at lists.openstack.org" <openstack at lists.openstack.org>,
> <Mail Attachment.gif>
> cc
> <Mail Attachment.gif>
> <Mail Attachment.gif>
> Subject
> <Mail Attachment.gif>
> [Openstack] [SWIFT] PUTs and GETs getting slower
>
>
> <Mail Attachment.gif> <Mail Attachment.gif>
> Hi,
>
> my Swift cluster is working for three weeks now as a mail storage and the
> performance is getting slower and slower.
>
> This is my hardware configuration:
>
> - 1 hardware-loadbalancer
> - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
> - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA
> 6GB/s
>
> The cluster contains:
>
> 279.000 containers
> 8.170.000 objects = 984 GByte
>
> I have about 3.000.000 GETs per day and 1.500.000 PUTs.
>
> The load is raising up to 12 now:
>
> top - 11:06:39 up 114 days, 23:10,  1 user,  load average: 10.88, 11.26,
> 11.22
> Tasks: 175 total,   3 running, 172 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 15.6 us,  9.8 sy,  0.0 ni, 69.3 id,  4.5 wa,  0.0 hi,  0.8 si,
>  0.0 st
> KiB Mem:  16386408 total, 14829360 used,  1557048 free,    30796 buffers
> KiB Swap: 17575932 total,   132604 used, 17443328 free,  5572360 cached
>
> PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
> 25880 swift     20   0 81400  15m 3352 D  34.5  0.1   2838:36
> /usr/bin/python /usr/bin/swift-container-replicator
> /etc/swift/container-server.conf
> 24282 swift     20   0 81084  14m 3244 S  19.2  0.1   3835:23
> /usr/bin/python /usr/bin/swift-container-auditor
> /etc/swift/container-server.conf
> 29816 swift     20   0 1523m  27m 2032 D   9.6  0.2   1132:58
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29812 swift     20   0 1523m  27m 2032 R   7.0  0.2   1169:50
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29815 swift     20   0 1523m  27m 2032 D   6.6  0.2   1132:45
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29809 swift     20   0 1523m  26m 2032 D   5.6  0.2   1163:53
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30684 swift     20   0 84496  16m 2468 D   5.6  0.1   1959:57
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 13853 snmp      20   0 51844 3356 1800 S   5.3  0.0  32:14.90
> /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p
> /var/run/snmpd.pid
> 29810 swift     20   0 1523m  27m 2032 D   5.3  0.2   1132:09
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29813 swift     20   0 1525m  27m 2032 D   5.3  0.2   1177:23
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 29811 swift     20   0 1524m  28m 2032 D   5.0  0.2   1169:37
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30677 swift     20   0 84932  17m 2468 D   5.0  0.1   1992:49
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 29814 swift     20   0 1524m  27m 2032 D   4.6  0.2   1181:11
> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
> 30681 swift     20   0 84980  17m 2468 S   4.3  0.1   1999:12
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30678 swift     20   0 84684  17m 2468 D   4.0  0.1   1973:14
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30682 swift     20   0 84224  16m 2468 S   4.0  0.1   1960:55
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30685 swift     20   0 84356  16m 2468 D   3.6  0.1   1979:40
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 7086 root      20   0     0    0    0 S   3.3  0.0   0:00.78 [kworker/6:1]
> 30683 swift     20   0 84160  16m 2468 S   3.3  0.1   1981:38
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 30679 swift     20   0 83852  16m 2468 S   3.0  0.1   1965:53
> /usr/bin/python /usr/bin/swift-container-server
> /etc/swift/container-server.conf
> 5935 root      20   0     0    0    0 S   2.7  0.0   0:16.38 [kworker/2:1]
> 6832 root      20   0     0    0    0 S   2.7  0.0   0:02.78 [kworker/4:0]
> 7153 swift     20   0 80300 9.8m 2040 D   2.7  0.1   0:00.08
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 1840 root      20   0     0    0    0 S   2.3  0.0   0:59.87 [kworker/0:1]
> 9630 swift     20   0 75796 6272 1708 D   2.0  0.0   1889:16
> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
> 9656 swift     20   0 87424  17m 2304 S   2.0  0.1 748:29.97
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9659 swift     20   0 86908  17m 2304 S   2.0  0.1 803:09.64
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9653 swift     20   0 85816  16m 2304 S   1.7  0.1 723:34.42
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9654 swift     20   0 87500  17m 2304 R   1.7  0.1 769:23.97
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9655 swift     20   0 85956  16m 2408 S   1.7  0.1 783:28.18
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9658 swift     20   0 87808  18m 2304 S   1.7  0.1 767:31.82
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 32616 root      20   0     0    0    0 S   1.7  0.0   0:18.13 [kworker/3:0]
> 1826 syslog    20   0  243m 5124  856 S   1.3  0.0 329:10.12 rsyslogd -c5
> 6944 root      20   0     0    0    0 S   1.3  0.0   0:01.08 [kworker/7:2]
> 9652 swift     20   0 86896  17m 2412 S   1.3  0.1 663:32.21
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 9657 swift     20   0 86016  16m 2408 S   1.3  0.1 706:22.08
> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
> 51 root      20   0     0    0    0 S   1.0  0.0 134:26.37 [kswapd0]
> 4548 root      20   0     0    0    0 S   1.0  0.0   0:11.69 [kworker/1:2]
> 5362 root      20   0     0    0    0 S   1.0  0.0   0:10.95 [kworker/5:3]
>  3 root      20   0     0    0    0 S   0.7  0.0 604:49.60 [ksoftirqd/0]
> 52 root      20   0     0    0    0 S   0.7  0.0  68:32.58 [kswapd1]
> 7412 swift     20   0 80300 9704 2780 S   0.7  0.1 566:00.53
> /usr/bin/python /usr/bin/swift-container-updater
> /etc/swift/container-server.conf
> 9580 swift     20   0 75632 7140 2728 D   0.7  0.0   2218:06
> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
> 22178 root      20   0  7404  992  700 D   0.7  0.0  20:28.93
> /usr/bin/updatedb.mlocate
>
> Is this a memory problem or a harddisk problem?
>
> And second question: On the server the mlocate database is rising.  Can I
> disable the updatedb.mlocate process and empty the database?
>
> Thanks
> Klaus
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
> Post to     :
> openstack at lists.openstack.org
>
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/54776f25/attachment.html>


More information about the Openstack mailing list