[Openstack] [SWIFT] PUTs and GETs getting slower

Hua ZZ Zhang zhuadl at cn.ibm.com
Mon Aug 5 10:35:05 UTC 2013


Here's the deployment guide which can give you options for tuning container
replicator and auditor, such as concurrency.
http://docs.openstack.org/developer/swift/deployment_guide.html
hope this can help you.

Best Regards,

                                                                             
 Edward Zhang(张华)                                                          
 Advisory Software Engineer                                                  
 Software Standards & Open Source                                            
 Software                                                                    
 Emerging Technology Institute(ETI)                                          
 IBM China Software Development Lab                                          
 e-mail: zhuadl at cn.ibm.com                                                   
                                                                             
                                                                             
                                                                             
                                                                             
                                                                             
                                                                             
                                                                             
                                                                             





                                                                           
             Klaus Schürmann                                               
             <klaus.schuermann                                             
             @mediabeam.com>                                            To 
                                       Hua ZZ Zhang/China/IBM at IBMCN,       
             08/05/2013 06:14                                           cc 
             PM                        "openstack at lists.openstack.org"     
                                       <openstack at lists.openstack.org>     
                                                                   Subject 
                                       AW: [Openstack] [SWIFT] PUTs and    
                                       GETs getting slower                 
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




Here is my container-server.conf:

[DEFAULT]
log_name = container-server
log_facility = LOG_LOCAL6
log_level = INFO
log_address = /dev/log
bind_ip = 10.12.1.1
workers = 8
log_statsd_host = 10.11.2.1
log_statsd_port = 8125
log_statsd_default_sample_rate = 0.25
log_statsd_metric_prefix = Storage01

[pipeline:main]
pipeline = recon container-server

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

[app:container-server]
use = egg:swift#container

[container-replicator]
concurrency = 2

[container-updater]
concurrency = 1

[container-auditor]

[container-sync]


--------


For the second question, Swift doesn't use  updatedb.mlocate in my mind. it
should be safe to disable it.
Based on the TOP output, it seems that the CPU workload of container
replicator and auditor are too high.
Perhaps you need to tune the config file to decrease their workload. paste
your config file of
container servers.

Best Regards,
                                                                             
       Edward Zhang(张华)                                                    
       Advisory Software Engineer                                            
       Software Standards & Open                                             
       Source Software                                                       
       Emerging Technology Institute                                         
       (ETI)                                                                 
       IBM China Software Development                                        
       Lab                                                                   
       e-mail: zhuadl at cn.ibm.com                                             
                                                                             
                                                                             
                                                                             
                                                                             
                                                                             



Inactive hide details for Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus
Schürmann <klaus.schuermann at mediabeam.com>Klaus Schürmann ---08/05/2013
05:43:08 PM---Klaus Schürmann <klaus.schuermann at mediabeam.com>


                                                                           
       Klaus Schürmann <                                                   
       klaus.schuermann at mediabeam.                                         
       com>                                                                
                                                                           
                                                                        To 
       08/05/2013 05:26 PM                                                 
                                                     "                     
                                                     openstack at lists.opens 
                                                     tack.org" <           
                                                     openstack at lists.opens 
                                                     tack.org>,            
                                                                           
                                                                        cc 
                                                                           
                                                                           
                                                                   Subject 
                                                                           
                                                     [Openstack] [SWIFT]   
                                                     PUTs and GETs getting 
                                                     slower                
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           



Hi,

my Swift cluster is working for three weeks now as a mail storage and the
performance is getting slower and slower.

This is my hardware configuration:

- 1 hardware-loadbalancer
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA
6GB/s

The cluster contains:

279.000 containers
8.170.000 objects = 984 GByte

I have about 3.000.000 GETs per day and 1.500.000 PUTs.

The load is raising up to 12 now:

top - 11:06:39 up 114 days, 23:10,  1 user,  load average: 10.88, 11.26,
11.22
Tasks: 175 total,   3 running, 172 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.6 us,  9.8 sy,  0.0 ni, 69.3 id,  4.5 wa,  0.0 hi,  0.8 si,
0.0 st
KiB Mem:  16386408 total, 14829360 used,  1557048 free,    30796 buffers
KiB Swap: 17575932 total,   132604 used, 17443328 free,  5572360 cached

 PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
25880 swift     20   0 81400  15m 3352 D  34.5  0.1
2838:36 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf

24282 swift     20   0 81084  14m 3244 S  19.2  0.1
3835:23 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf

29816 swift     20   0 1523m  27m 2032 D   9.6  0.2
1132:58 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

29812 swift     20   0 1523m  27m 2032 R   7.0  0.2
1169:50 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

29815 swift     20   0 1523m  27m 2032 D   6.6  0.2
1132:45 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

29809 swift     20   0 1523m  26m 2032 D   5.6  0.2
1163:53 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

30684 swift     20   0 84496  16m 2468 D   5.6  0.1
1959:57 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

13853 snmp      20   0 51844 3356 1800 S   5.3  0.0
32:14.90 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux
-p /var/run/snmpd.pid
29810 swift     20   0 1523m  27m 2032 D   5.3  0.2
1132:09 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

29813 swift     20   0 1525m  27m 2032 D   5.3  0.2
1177:23 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

29811 swift     20   0 1524m  28m 2032 D   5.0  0.2
1169:37 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

30677 swift     20   0 84932  17m 2468 D   5.0  0.1
1992:49 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

29814 swift     20   0 1524m  27m 2032 D   4.6  0.2
1181:11 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

30681 swift     20   0 84980  17m 2468 S   4.3  0.1
1999:12 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

30678 swift     20   0 84684  17m 2468 D   4.0  0.1
1973:14 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

30682 swift     20   0 84224  16m 2468 S   4.0  0.1
1960:55 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

30685 swift     20   0 84356  16m 2468 D   3.6  0.1
1979:40 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

7086 root      20   0     0    0    0 S   3.3  0.0   0:00.78 [kworker/6:1]
30683 swift     20   0 84160  16m 2468 S   3.3  0.1
1981:38 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

30679 swift     20   0 83852  16m 2468 S   3.0  0.1
1965:53 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

5935 root      20   0     0    0    0 S   2.7  0.0   0:16.38 [kworker/2:1]
6832 root      20   0     0    0    0 S   2.7  0.0   0:02.78 [kworker/4:0]
7153 swift     20   0 80300 9.8m 2040 D   2.7  0.1
0:00.08 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf

1840 root      20   0     0    0    0 S   2.3  0.0   0:59.87 [kworker/0:1]
9630 swift     20   0 75796 6272 1708 D   2.0  0.0
1889:16 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf

9656 swift     20   0 87424  17m 2304 S   2.0  0.1
748:29.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9659 swift     20   0 86908  17m 2304 S   2.0  0.1
803:09.64 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9653 swift     20   0 85816  16m 2304 S   1.7  0.1
723:34.42 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9654 swift     20   0 87500  17m 2304 R   1.7  0.1
769:23.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9655 swift     20   0 85956  16m 2408 S   1.7  0.1
783:28.18 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9658 swift     20   0 87808  18m 2304 S   1.7  0.1
767:31.82 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

32616 root      20   0     0    0    0 S   1.7  0.0   0:18.13 [kworker/3:0]
1826 syslog    20   0  243m 5124  856 S   1.3  0.0 329:10.12 rsyslogd -c5
6944 root      20   0     0    0    0 S   1.3  0.0   0:01.08 [kworker/7:2]
9652 swift     20   0 86896  17m 2412 S   1.3  0.1
663:32.21 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

9657 swift     20   0 86016  16m 2408 S   1.3  0.1
706:22.08 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

  51 root      20   0     0    0    0 S   1.0  0.0 134:26.37 [kswapd0]
4548 root      20   0     0    0    0 S   1.0  0.0   0:11.69 [kworker/1:2]
5362 root      20   0     0    0    0 S   1.0  0.0   0:10.95 [kworker/5:3]
   3 root      20   0     0    0    0 S   0.7  0.0 604:49.60 [ksoftirqd/0]
  52 root      20   0     0    0    0 S   0.7  0.0  68:32.58 [kswapd1]
7412 swift     20   0 80300 9704 2780 S   0.7  0.1
566:00.53 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf

9580 swift     20   0 75632 7140 2728 D   0.7  0.0
2218:06 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf

22178 root      20   0  7404  992  700 D   0.7  0.0
20:28.93 /usr/bin/updatedb.mlocate

Is this a memory problem or a harddisk problem?

And second question: On the server the mlocate database is rising.  Can I
disable the updatedb.mlocate process and empty the database?

Thanks
Klaus



_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13844866.gif
Type: image/gif
Size: 1279 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic22692.gif
Type: image/gif
Size: 1255 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment-0003.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13813793.gif
Type: image/gif
Size: 166 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13823512.gif
Type: image/gif
Size: 168 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130805/f80e1546/attachment-0005.gif>


More information about the Openstack mailing list