[Openstack] [SWIFT] Bad replication performance after adding new drives
Klaus Schürmann
klaus.schuermann at mediabeam.com
Tue Feb 10 09:44:22 UTC 2015
Hi,
I have a swift cluster with 7 storagenodes. The cluster has about 360 million objects with an average size of 170kB.
I added 2 new harddrives to my cluster and now it takes weeks for one replication round:
Feb 10 09:40:41 storage-node1 object-replicator: 7698658 suffixes checked - 9.03% hashed, 23.07% synced
Feb 10 09:40:41 storage-node1 object-replicator: Partition times: max 9046.7893s, min 0.0908s, med 199.9172s
Feb 10 09:45:41 storage-node1 object-replicator: 1945/9082 (21.42%) partitions replicated in 525611.08s (0.00/sec, 535h remaining)
Is this a normal behavior?
SWIFT
Version: 1.13.1
Hardware:
DELL PowerEdge R720xd
32 GB RAM
Harddisks 3TB/4TB SATA
The account-server and container-server are stored on additional SSDs on all 7 servers.
object-server.conf:
[object-replicator]
concurrency = 3
slabtop:
Active / Total Objects (% used) : 31740287 / 35050641 (90.6%)
Active / Total Slabs (% used) : 1037707 / 1037707 (100.0%)
Active / Total Caches (% used) : 71 / 111 (64.0%)
Active / Total Size (% used) : 13271033.09K / 14170940.08K (93.6%)
Minimum / Average / Maximum Object : 0.01K / 0.40K / 15.23K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
9007808 8167956 90% 0.06K 140747 64 562988K kmalloc-64
8792592 8666410 98% 0.88K 391604 36 12531328K xfs_inode
6447665 5790740 89% 0.55K 230564 28 3689024K radix_tree_node
4763220 4324615 90% 0.19K 113410 42 907280K dentry
3092256 2386211 77% 0.22K 85896 36 687168K xfs_buf_item
1003767 752404 74% 0.31K 19879 51 318064K ip6_dst_cache
593632 457963 77% 0.12K 18551 32 74204K kmalloc-128
507546 466652 91% 0.10K 13014 39 52056K buffer_head
iotop:
Total DISK READ: 23.87 M/s | Total DISK WRITE: 24.06 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
23239 be/4 swift 676.34 K/s 0.00 B/s 0.00 % 21.72 % rsync --recursive --whole-file --~3715/056 /srv/node/sde1/objects/3
12691 be/4 swift 12.81 M/s 51.18 K/s 0.00 % 19.85 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12699 be/4 swift 80.43 K/s 106.02 K/s 0.00 % 13.87 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
22058 be/4 swift 456.98 K/s 0.00 B/s 0.00 % 12.92 % rsync --recursive --whole-file --~4057/29b /srv/node/sdc1/objects/4
12687 be/4 swift 95.05 K/s 58.49 K/s 0.00 % 8.97 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12962 be/4 swift 120.64 K/s 120.64 K/s 0.00 % 7.40 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
2330 be/4 swift 453.33 K/s 0.00 B/s 0.00 % 7.27 % python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
12693 be/4 swift 146.23 K/s 14.62 K/s 0.00 % 6.14 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12701 be/4 swift 40.21 K/s 0.00 B/s 0.00 % 5.99 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12694 be/4 swift 84.08 K/s 25.59 K/s 0.00 % 5.13 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12700 be/4 swift 51.18 K/s 0.00 B/s 0.00 % 4.37 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12690 be/4 swift 153.55 K/s 62.15 K/s 0.00 % 4.03 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12682 be/4 swift 109.68 K/s 14.62 K/s 0.00 % 3.75 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12689 be/4 swift 69.46 K/s 65.81 K/s 0.00 % 3.13 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
13095 be/4 swift 146.23 K/s 124.30 K/s 0.00 % 3.11 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12376 be/4 swift 1297.83 K/s 21.94 K/s 0.00 % 2.99 % python /usr/bin/swift-container-r~ /etc/swift/container-server.conf
1556 be/4 root 2.82 M/s 0.00 B/s 0.00 % 2.87 % [xfsaild/sdd1]
2270 be/4 swift 592.25 K/s 76.77 K/s 0.00 % 2.83 % python /usr/bin/swift-container-a~ /etc/swift/container-server.conf
12697 be/4 swift 29.25 K/s 0.00 B/s 0.00 % 2.61 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12698 be/4 swift 153.55 K/s 168.17 K/s 0.00 % 2.52 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12686 be/4 swift 7.31 K/s 0.00 B/s 0.00 % 2.29 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
2341 be/4 swift 142.58 K/s 80.43 K/s 0.00 % 2.22 % python /usr/bin/swift-container-s~ /etc/swift/container-server.conf
12683 be/4 swift 25.59 K/s 0.00 B/s 0.00 % 1.93 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12745 be/4 swift 127.96 K/s 0.00 B/s 0.00 % 1.55 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12696 be/4 swift 76.77 K/s 95.05 K/s 0.00 % 1.53 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12685 be/4 swift 36.56 K/s 51.18 K/s 0.00 % 1.50 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
12688 be/4 swift 58.49 K/s 18.28 K/s 0.00 % 0.84 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
2342 be/4 swift 98.71 K/s 58.49 K/s 0.00 % 0.65 % python /usr/bin/swift-container-s~ /etc/swift/container-server.conf
2344 be/4 swift 43.87 K/s 40.21 K/s 0.00 % 0.65 % python /usr/bin/swift-container-s~ /etc/swift/container-server.conf
12861 be/4 swift 153.55 K/s 138.92 K/s 0.00 % 0.57 % python /usr/bin/swift-object-server /etc/swift/object-server.conf
2340 be/4 swift 14.62 K/s 73.12 K/s 0.00 % 0.52 % python /usr/bin/swift-container-s~ /etc/swift/container-server.conf
swift-ringbuilder output for the object ring:
object.builder, build version 174
16384 partitions, 3.000000 replicas, 1 regions, 7 zones, 63 devices, 0.98 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
0 1 1 10.12.1.1 6000 10.12.1.1 6000 sda4 3000.00 630 -0.02
1 1 2 10.12.1.2 6000 10.12.1.2 6000 sda4 3000.00 630 -0.02
2 1 3 10.12.1.3 6000 10.12.1.3 6000 sda4 3000.00 631 0.13
3 1 4 10.12.1.4 6000 10.12.1.4 6000 sda4 3000.00 630 -0.02
4 1 5 10.12.1.5 6000 10.12.1.5 6000 sda4 3000.00 631 0.13
5 1 1 10.12.1.1 6000 10.12.1.1 6000 sdb1 3000.00 630 -0.02
6 1 2 10.12.1.2 6000 10.12.1.2 6000 sdb1 3000.00 630 -0.02
7 1 3 10.12.1.3 6000 10.12.1.3 6000 sdb1 3000.00 631 0.13
8 1 4 10.12.1.4 6000 10.12.1.4 6000 sdb1 3000.00 630 -0.02
9 1 5 10.12.1.5 6000 10.12.1.5 6000 sdb1 3000.00 631 0.13
10 1 1 10.12.1.1 6000 10.12.1.1 6000 sdc1 4000.00 840 -0.02
11 1 2 10.12.1.2 6000 10.12.1.2 6000 sdc1 4000.00 841 0.09
12 1 3 10.12.1.3 6000 10.12.1.3 6000 sdc1 4000.00 841 0.09
13 1 4 10.12.1.4 6000 10.12.1.4 6000 sdc1 4000.00 841 0.09
14 1 5 10.12.1.5 6000 10.12.1.5 6000 sdc1 4000.00 840 -0.02
15 1 1 10.12.1.1 6000 10.12.1.1 6000 sde1 4000.00 840 -0.02
16 1 2 10.12.1.2 6000 10.12.1.2 6000 sde1 4000.00 841 0.09
17 1 3 10.12.1.3 6000 10.12.1.3 6000 sde1 4000.00 840 -0.02
18 1 4 10.12.1.4 6000 10.12.1.4 6000 sde1 4000.00 840 -0.02
19 1 5 10.12.1.5 6000 10.12.1.5 6000 sde1 4000.00 841 0.09
20 1 1 10.12.1.1 6000 10.12.1.1 6000 sdf1 4000.00 840 -0.02
21 1 2 10.12.1.2 6000 10.12.1.2 6000 sdf1 4000.00 841 0.09
22 1 3 10.12.1.3 6000 10.12.1.3 6000 sdf1 4000.00 840 -0.02
23 1 4 10.12.1.4 6000 10.12.1.4 6000 sdf1 4000.00 840 -0.02
24 1 5 10.12.1.5 6000 10.12.1.5 6000 sdf1 4000.00 840 -0.02
25 1 1 10.12.1.1 6000 10.12.1.1 6000 sdg1 4000.00 840 -0.02
26 1 2 10.12.1.2 6000 10.12.1.2 6000 sdg1 4000.00 841 0.09
27 1 3 10.12.1.3 6000 10.12.1.3 6000 sdg1 4000.00 840 -0.02
28 1 4 10.12.1.4 6000 10.12.1.4 6000 sdg1 4000.00 840 -0.02
29 1 5 10.12.1.5 6000 10.12.1.5 6000 sdg1 4000.00 840 -0.02
30 1 1 10.12.1.1 6000 10.12.1.1 6000 sdh1 4000.00 840 -0.02
31 1 2 10.12.1.2 6000 10.12.1.2 6000 sdh1 4000.00 840 -0.02
32 1 3 10.12.1.3 6000 10.12.1.3 6000 sdh1 4000.00 841 0.09
33 1 4 10.12.1.4 6000 10.12.1.4 6000 sdh1 4000.00 841 0.09
34 1 5 10.12.1.5 6000 10.12.1.5 6000 sdh1 4000.00 840 -0.02
35 1 1 10.12.1.1 6000 10.12.1.1 6000 sdi1 4000.00 840 -0.02
36 1 2 10.12.1.2 6000 10.12.1.2 6000 sdi1 4000.00 841 0.09
37 1 3 10.12.1.3 6000 10.12.1.3 6000 sdi1 4000.00 841 0.09
38 1 4 10.12.1.4 6000 10.12.1.4 6000 sdi1 4000.00 841 0.09
39 1 5 10.12.1.5 6000 10.12.1.5 6000 sdi1 4000.00 840 -0.02
40 1 1 10.12.1.1 6000 10.12.1.1 6000 sdj1 4000.00 840 -0.02
41 1 2 10.12.1.2 6000 10.12.1.2 6000 sdj1 4000.00 840 -0.02
42 1 3 10.12.1.3 6000 10.12.1.3 6000 sdj1 4000.00 840 -0.02
43 1 4 10.12.1.4 6000 10.12.1.4 6000 sdj1 4000.00 841 0.09
44 1 5 10.12.1.5 6000 10.12.1.5 6000 sdj1 4000.00 841 0.09
45 1 1 10.12.1.1 6000 10.12.1.1 6000 sdk1 4000.00 840 -0.02
46 1 2 10.12.1.2 6000 10.12.1.2 6000 sdk1 4000.00 841 0.09
47 1 3 10.12.1.3 6000 10.12.1.3 6000 sdk1 4000.00 841 0.09
48 1 4 10.12.1.4 6000 10.12.1.4 6000 sdk1 4000.00 841 0.09
49 1 5 10.12.1.5 6000 10.12.1.5 6000 sdk1 4000.00 840 -0.02
50 1 2 10.12.1.2 6000 10.12.1.2 6000 sdl1 4000.00 841 0.09
51 1 3 10.12.1.3 6000 10.12.1.3 6000 sdl1 4000.00 840 -0.02
52 1 4 10.12.1.4 6000 10.12.1.4 6000 sdl1 4000.00 840 -0.02
53 1 5 10.12.1.5 6000 10.12.1.5 6000 sdl1 4000.00 841 0.09
54 1 2 10.12.1.1 6000 10.12.1.1 6000 sdl1 4000.00 840 -0.02
55 1 6 10.12.1.6 6000 10.12.1.6 6000 sda1 3000.00 630 -0.02
56 1 7 10.12.1.7 6000 10.12.1.7 6000 sda1 3000.00 630 -0.02
57 1 1 10.12.1.6 6000 10.12.1.6 6000 sdb1 3000.00 630 -0.02
58 1 1 10.12.1.7 6000 10.12.1.7 6000 sdb1 3000.00 630 -0.02
59 1 1 10.12.1.6 6000 10.12.1.6 6000 sdc1 3000.00 630 -0.02
60 1 1 10.12.1.7 6000 10.12.1.7 6000 sdc1 3000.00 630 -0.02
61 1 1 10.12.1.6 6000 10.12.1.6 6000 sdd1 3000.00 625 -0.82
62 1 1 10.12.1.7 6000 10.12.1.7 6000 sdd1 3000.00 624 -0.98
More information about the Openstack
mailing list