On Mon, Nov 13, 2023 at 2:39 PM <mgutha@liveperson.com> wrote:

I have observed an imbalance in the cluster based on the object.builder output. Is this possibly due to an ongoing rebalance process?

Additionally, I'd like to confirm my understanding of 'min_part_hours.' Is it accurate to say that this parameter represents the number of hours an object needs to reside on a disk before it's considered for movement to another device during a rebalance? According to the Swift documentation, the default value is 24 hours. I'm curious if this value is applied to data written specifically by the rebalance process or if it also includes data written by other Swift services.

object.builder, build version 775, id 61b2166cb8684b2b80d6fc370cef37a6
65536 partitions, 2.000000 replicas, 2 regions, 2 zones, 95 devices, 38.89 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining)
The overload factor is 10.00% (0.100000)
Ring file object.ring.gz not found, probably it hasn't been written yet
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
129 1 1 182.16.128.36:6000 182.16.128.36:6003 d129 1.00 2 38.89
130 1 1 182.16.128.36:6006 182.16.128.36:6003 d130 1.00 2 38.89
131 1 1 182.16.128.36:6007 182.16.128.36:6003 d131 1.00 2 38.89
132 1 1 182.16.128.36:6008 182.16.128.36:6003 d132 1.00 2 38.89
133 1 1 182.16.128.36:6009 182.16.128.36:6003 d133 1.00 2 38.89
134 1 1 182.16.128.36:6010 182.16.128.36:6003 d134 1.00 1 -30.56
135 1 1 182.16.128.36:6011 182.16.128.36:6003 d135 1.00 1 -30.56
136 1 1 182.16.128.36:6012 182.16.128.36:6003 d136 1.00 1 -30.56
137 1 1 182.16.128.36:6013 182.16.128.36:6003 d137 1.00 1 -30.56
138 1 1 182.16.128.36:6014 182.16.128.36:6003 d138 1.00 1 -30.56
139 1 1 182.16.128.36:6015 182.16.128.36:6003 d139 1.00 1 -30.56
140 1 1 182.16.128.38:6000 182.16.128.38:6003 d140 1.00 2 38.89
141 1 1 182.16.128.38:6006 182.16.128.38:6003 d141 1.00 2 38.89
142 1 1 182.16.128.38:6007 182.16.128.38:6003 d142 1.00 2 38.89
143 1 1 182.16.128.38:6008 182.16.128.38:6003 d143 1.00 2 38.89
144 1 1 182.16.128.38:6009 182.16.128.38:6003 d144 1.00 2 38.89
145 1 1 182.16.128.38:6010 182.16.128.38:6003 d145 1.00 1 -30.56
146 1 1 182.16.128.38:6011 182.16.128.38:6003 d146 1.00 1 -30.56
147 1 1 182.16.128.38:6012 182.16.128.38:6003 d147 1.00 1 -30.56
148 1 1 182.16.128.38:6013 182.16.128.38:6003 d148 1.00 1 -30.56
149 1 1 182.16.128.38:6014 182.16.128.38:6003 d149 1.00 1 -30.56
150 1 1 182.16.128.38:6015 182.16.128.38:6003 d150 1.00 1 -30.56
151 1 1 182.16.128.39:6000 182.16.128.39:6003 d151 1.00 2 38.89
152 1 1 182.16.128.39:6006 182.16.128.39:6003 d152 1.00 2 38.89
153 1 1 182.16.128.39:6007 182.16.128.39:6003 d153 1.00 2 38.89
154 1 1 182.16.128.39:6008 182.16.128.39:6003 d154 1.00 2 38.89
155 1 1 182.16.128.39:6009 182.16.128.39:6003 d155 1.00 2 38.89
156 1 1 182.16.128.39:6010 182.16.128.39:6003 d156 1.00 1 -30.56
157 1 1 182.16.128.39:6011 182.16.128.39:6003 d157 1.00 1 -30.56
158 1 1 182.16.128.39:6012 182.16.128.39:6003 d158 1.00 1 -30.56
159 1 1 182.16.128.39:6013 182.16.128.39:6003 d159 1.00 1 -30.56
160 1 1 182.16.128.39:6014 182.16.128.39:6003 d160 1.00 1 -30.56
161 1 1 182.16.128.39:6015 182.16.128.39:6003 d161 1.00 1 -30.56
55 1 1 182.16.128.5:6000 182.16.128.5:6003 d55 2000.40 2789 -3.18
56 1 1 182.16.128.5:6006 182.16.128.5:6003 d56 2000.40 2789 -3.18
57 1 1 182.16.128.5:6007 182.16.128.5:6003 d57 2000.40 2789 -3.18
58 1 1 182.16.128.5:6008 182.16.128.5:6003 d58 2000.40 2789 -3.18
59 1 1 182.16.128.5:6009 182.16.128.5:6003 d59 2000.40 2789 -3.18
60 1 1 182.16.128.5:6010 182.16.128.5:6003 d60 2000.40 2789 -3.18
61 1 1 182.16.128.5:6011 182.16.128.5:6003 d61 2000.40 2789 -3.18
62 1 1 182.16.128.5:6012 182.16.128.5:6003 d62 2000.40 2789 -3.18
63 1 1 182.16.128.5:6013 182.16.128.5:6003 d63 2000.40 2789 -3.18
64 1 1 182.16.128.5:6014 182.16.128.5:6003 d64 2000.40 2789 -3.18
65 1 1 182.16.128.5:6015 182.16.128.5:6003 d65 2000.40 2788 -3.21
44 1 1 182.16.128.6:6000 182.16.128.6:6003 d44 269.43 376 -3.09
45 1 1 182.16.128.6:6006 182.16.128.6:6003 d45 269.43 376 -3.09
46 1 1 182.16.128.6:6007 182.16.128.6:6003 d46 269.43 376 -3.09
47 1 1 182.16.128.6:6008 182.16.128.6:6003 d47 269.43 376 -3.09
48 1 1 182.16.128.6:6009 182.16.128.6:6003 d48 269.43 376 -3.09
49 1 1 182.16.128.6:6010 182.16.128.6:6003 d49 269.43 376 -3.09
50 1 1 182.16.128.6:6011 182.16.128.6:6003 d50 269.43 376 -3.09
51 1 1 182.16.128.6:6012 182.16.128.6:6003 d51 269.43 375 -3.35
52 1 1 182.16.128.6:6013 182.16.128.6:6003 d52 269.43 375 -3.35
53 1 1 182.16.128.6:6014 182.16.128.6:6003 d53 269.43 375 -3.35
54 1 1 182.16.128.6:6015 182.16.128.6:6003 d54 269.43 375 -3.35
0 1 1 182.16.128.7:6000 182.16.128.7:6003 d0 2000.40 2789 -3.18
1 1 1 182.16.128.7:6006 182.16.128.7:6003 d1 2000.40 2789 -3.18
10 1 1 182.16.128.7:6015 182.16.128.7:6003 d10 2000.40 2788 -3.21
2 1 1 182.16.128.7:6007 182.16.128.7:6003 d2 2000.40 2789 -3.18
3 1 1 182.16.128.7:6008 182.16.128.7:6003 d3 2000.40 2789 -3.18
4 1 1 182.16.128.7:6009 182.16.128.7:6003 d4 2000.40 2789 -3.18
5 1 1 182.16.128.7:6010 182.16.128.7:6003 d5 2000.40 2789 -3.18
6 1 1 182.16.128.7:6011 182.16.128.7:6003 d6 2000.40 2789 -3.18
7 1 1 182.16.128.7:6012 182.16.128.7:6003 d7 2000.40 2789 -3.18
8 1 1 182.16.128.7:6013 182.16.128.7:6003 d8 2000.40 2789 -3.18
9 1 1 182.16.128.7:6014 182.16.128.7:6003 d9 2000.40 2789 -3.18
11 2 2 11.17.128.5:6000 11.17.128.5:6003 d11 2000.40 2979 3.42
12 2 2 11.17.128.5:6006 11.17.128.5:6003 d12 2000.40 2979 3.42
13 2 2 11.17.128.5:6007 11.17.128.5:6003 d13 2000.40 2979 3.42
15 2 2 11.17.128.5:6008 11.17.128.5:6003 d15 2000.40 2979 3.42
16 2 2 11.17.128.5:6009 11.17.128.5:6003 d16 2000.40 2979 3.42
17 2 2 11.17.128.5:6010 11.17.128.5:6003 d17 2000.40 2978 3.38
18 2 2 11.17.128.5:6011 11.17.128.5:6003 d18 2000.40 2978 3.38
19 2 2 11.17.128.5:6012 11.17.128.5:6003 d19 2000.40 2978 3.38
20 2 2 11.17.128.5:6013 11.17.128.5:6003 d20 2000.40 2978 3.38
21 2 2 11.17.128.5:6014 11.17.128.5:6003 d21 2000.40 2978 3.38
77 2 2 11.17.128.5:6015 11.17.128.5:6003 d77 2000.40 2978 3.38
33 2 2 11.17.128.6:6000 11.17.128.6:6003 d33 2000.40 2979 3.42
34 2 2 11.17.128.6:6006 11.17.128.6:6003 d34 2000.40 2979 3.42
35 2 2 11.17.128.6:6007 11.17.128.6:6003 d35 2000.40 2979 3.42
36 2 2 11.17.128.6:6008 11.17.128.6:6003 d36 2000.40 2979 3.42
37 2 2 11.17.128.6:6009 11.17.128.6:6003 d37 2000.40 2978 3.38
38 2 2 11.17.128.6:6010 11.17.128.6:6003 d38 2000.40 2978 3.38
39 2 2 11.17.128.6:6011 11.17.128.6:6003 d39 2000.40 2978 3.38
40 2 2 11.17.128.6:6012 11.17.128.6:6003 d40 2000.40 2978 3.38
41 2 2 11.17.128.6:6013 11.17.128.6:6003 d41 2000.40 2978 3.38
42 2 2 11.17.128.6:6014 11.17.128.6:6003 d42 2000.40 2978 3.38
43 2 2 11.17.128.6:6015 11.17.128.6:6003 d43 2000.40 2978 3.38
162 2 2 11.17.128.71:6000 11.17.128.71:6003 d162 1.00 2 38.89
163 2 2 11.17.128.71:6006 11.17.128.71:6003 d163 1.00 2 38.89
164 2 2 11.17.128.71:6007 11.17.128.71:6003 d164 1.00 2 38.89
165 2 2 11.17.128.71:6008 11.17.128.71:6003 d165 1.00 2 38.89
166 2 2 11.17.128.71:6009 11.17.128.71:6003 d166 1.00 1 -30.56
167 2 2 11.17.128.71:6010 11.17.128.71:6003 d167 1.00 1 -30.56
168 2 2 11.17.128.71:6011 11.17.128.71:6003 d168 1.00 1 -30.56