Open Stack

Hi,

my Swift cluster is working for three weeks now as a mail storage and the performance is getting slower and slower.

This is my hardware configuration:

- 1 hardware-loadbalancer
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA 6GB/s

The cluster contains:

279.000 containers
8.170.000 objects = 984 GByte

I have about 3.000.000 GETs per day and 1.500.000 PUTs.

The load is raising up to 12 now:

top - 11:06:39 up 114 days, 23:10,  1 user,  load average: 10.88, 11.26, 11.22
Tasks: 175 total,   3 running, 172 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.6 us,  9.8 sy,  0.0 ni, 69.3 id,  4.5 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem:  16386408 total, 14829360 used,  1557048 free,    30796 buffers
KiB Swap: 17575932 total,   132604 used, 17443328 free,  5572360 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
25880 swift     20   0 81400  15m 3352 D  34.5  0.1   2838:36 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf
24282 swift     20   0 81084  14m 3244 S  19.2  0.1   3835:23 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf
29816 swift     20   0 1523m  27m 2032 D   9.6  0.2   1132:58 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29812 swift     20   0 1523m  27m 2032 R   7.0  0.2   1169:50 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29815 swift     20   0 1523m  27m 2032 D   6.6  0.2   1132:45 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29809 swift     20   0 1523m  26m 2032 D   5.6  0.2   1163:53 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30684 swift     20   0 84496  16m 2468 D   5.6  0.1   1959:57 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
13853 snmp      20   0 51844 3356 1800 S   5.3  0.0  32:14.90 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid
29810 swift     20   0 1523m  27m 2032 D   5.3  0.2   1132:09 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29813 swift     20   0 1525m  27m 2032 D   5.3  0.2   1177:23 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
29811 swift     20   0 1524m  28m 2032 D   5.0  0.2   1169:37 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30677 swift     20   0 84932  17m 2468 D   5.0  0.1   1992:49 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
29814 swift     20   0 1524m  27m 2032 D   4.6  0.2   1181:11 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
30681 swift     20   0 84980  17m 2468 S   4.3  0.1   1999:12 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30678 swift     20   0 84684  17m 2468 D   4.0  0.1   1973:14 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30682 swift     20   0 84224  16m 2468 S   4.0  0.1   1960:55 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30685 swift     20   0 84356  16m 2468 D   3.6  0.1   1979:40 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
 7086 root      20   0     0    0    0 S   3.3  0.0   0:00.78 [kworker/6:1]
30683 swift     20   0 84160  16m 2468 S   3.3  0.1   1981:38 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
30679 swift     20   0 83852  16m 2468 S   3.0  0.1   1965:53 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
 5935 root      20   0     0    0    0 S   2.7  0.0   0:16.38 [kworker/2:1]
 6832 root      20   0     0    0    0 S   2.7  0.0   0:02.78 [kworker/4:0]
 7153 swift     20   0 80300 9.8m 2040 D   2.7  0.1   0:00.08 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf
 1840 root      20   0     0    0    0 S   2.3  0.0   0:59.87 [kworker/0:1]
 9630 swift     20   0 75796 6272 1708 D   2.0  0.0   1889:16 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
 9656 swift     20   0 87424  17m 2304 S   2.0  0.1 748:29.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9659 swift     20   0 86908  17m 2304 S   2.0  0.1 803:09.64 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9653 swift     20   0 85816  16m 2304 S   1.7  0.1 723:34.42 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9654 swift     20   0 87500  17m 2304 R   1.7  0.1 769:23.97 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9655 swift     20   0 85956  16m 2408 S   1.7  0.1 783:28.18 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9658 swift     20   0 87808  18m 2304 S   1.7  0.1 767:31.82 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
32616 root      20   0     0    0    0 S   1.7  0.0   0:18.13 [kworker/3:0]
 1826 syslog    20   0  243m 5124  856 S   1.3  0.0 329:10.12 rsyslogd -c5
 6944 root      20   0     0    0    0 S   1.3  0.0   0:01.08 [kworker/7:2]
 9652 swift     20   0 86896  17m 2412 S   1.3  0.1 663:32.21 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
 9657 swift     20   0 86016  16m 2408 S   1.3  0.1 706:22.08 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
   51 root      20   0     0    0    0 S   1.0  0.0 134:26.37 [kswapd0]
 4548 root      20   0     0    0    0 S   1.0  0.0   0:11.69 [kworker/1:2]
 5362 root      20   0     0    0    0 S   1.0  0.0   0:10.95 [kworker/5:3]
    3 root      20   0     0    0    0 S   0.7  0.0 604:49.60 [ksoftirqd/0]
   52 root      20   0     0    0    0 S   0.7  0.0  68:32.58 [kswapd1]
 7412 swift     20   0 80300 9704 2780 S   0.7  0.1 566:00.53 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf
 9580 swift     20   0 75632 7140 2728 D   0.7  0.0   2218:06 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
22178 root      20   0  7404  992  700 D   0.7  0.0  20:28.93 /usr/bin/updatedb.mlocate

Is this a memory problem or a harddisk problem?

And second question: On the server the mlocate database is rising.  Can I disable the updatedb.mlocate process and empty the database?

Thanks
Klaus