We have two Regions; VA(r1z1) /CA (r2z2) when we attempted to remove the CA nodes from cluster we started to see the VA storage shoot up due to the twoperregion replication policy. As i mentioned earlier we have started the CA nodes back up and also added 3 more nodes with (10 drivers per node) to existing VA cluster what we are seeing is that rebalance is not distributing the data to newly added hosts.
va-newnode-0[1-3] are the new nodes that were added to VA DC (r1z1) region, and also one the old va node va-oldnode-02 is doesn't have the data evenly distributed. va-oldnode-02 had a drive failures and we have added the node back since then the rebalance has not yet completed.
object.builder, build version 775, id 61b2166cb8684b2b80d6fc370cef37a6 65536 partitions, 2.000000 replicas, 2 regions, 2 zones, 95 devices, 38.89 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining) The overload factor is 10.00% (0.100000) Ring file object.ring.gz not found, probably it hasn't been written yet Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 129 1 1 va-newnode-01:6000 va-newip-01:6003 d129 1.00 2 38.89 130 1 1 va-newnode-01:6006 va-newip-01:6003 d130 1.00 2 38.89 131 1 1 va-newnode-01:6007 va-newip-01:6003 d131 1.00 2 38.89 132 1 1 va-newnode-01:6008 va-newip-01:6003 d132 1.00 2 38.89 133 1 1 va-newnode-01:6009 va-newip-01:6003 d133 1.00 2 38.89 134 1 1 va-newnode-01:6010 va-newip-01:6003 d134 1.00 1 -30.56 135 1 1 va-newnode-01:6011 va-newip-01:6003 d135 1.00 1 -30.56 136 1 1 va-newnode-01:6012 va-newip-01:6003 d136 1.00 1 -30.56 137 1 1 va-newnode-01:6013 va-newip-01:6003 d137 1.00 1 -30.56 138 1 1 va-newnode-01:6014 va-newip-01:6003 d138 1.00 1 -30.56 139 1 1 va-newnode-01:6015 va-newip-01:6003 d139 1.00 1 -30.56 140 1 1 va-newnode-02:6000 va-newnode-02:6003 d140 1.00 2 38.89 141 1 1 va-newnode-02:6006 va-newnode-02:6003 d141 1.00 2 38.89 142 1 1 va-newnode-02:6007 va-newnode-02:6003 d142 1.00 2 38.89 143 1 1 va-newnode-02:6008 va-newnode-02:6003 d143 1.00 2 38.89 144 1 1 va-newnode-02:6009 va-newnode-02:6003 d144 1.00 2 38.89 145 1 1 va-newnode-02:6010 va-newnode-02:6003 d145 1.00 1 -30.56 146 1 1 va-newnode-02:6011 va-newnode-02:6003 d146 1.00 1 -30.56 147 1 1 va-newnode-02:6012 va-newnode-02:6003 d147 1.00 1 -30.56 148 1 1 va-newnode-02:6013 va-newnode-02:6003 d148 1.00 1 -30.56 149 1 1 va-newnode-02:6014 va-newnode-02:6003 d149 1.00 1 -30.56 150 1 1 va-newnode-02:6015 va-newnode-02:6003 d150 1.00 1 -30.56 151 1 1 va-newnode-03:6000 va-newnode-03:6003 d151 1.00 2 38.89 152 1 1 va-newnode-03:6006 va-newnode-03:6003 d152 1.00 2 38.89 153 1 1 va-newnode-03:6007 va-newnode-03:6003 d153 1.00 2 38.89 154 1 1 va-newnode-03:6008 va-newnode-03:6003 d154 1.00 2 38.89 155 1 1 va-newnode-03:6009 va-newnode-03:6003 d155 1.00 2 38.89 156 1 1 va-newnode-03:6010 va-newnode-03:6003 d156 1.00 1 -30.56 157 1 1 va-newnode-03:6011 va-newnode-03:6003 d157 1.00 1 -30.56 158 1 1 va-newnode-03:6012 va-newnode-03:6003 d158 1.00 1 -30.56 159 1 1 va-newnode-03:6013 va-newnode-03:6003 d159 1.00 1 -30.56 160 1 1 va-newnode-03:6014 va-newnode-03:6003 d160 1.00 1 -30.56 161 1 1 va-newnode-03:6015 va-newnode-03:6003 d161 1.00 1 -30.56 55 1 1 va-oldnode-01:6000 va-oldnode-01:6003 d55 2000.40 2789 -3.18 56 1 1 va-oldnode-01:6006 va-oldnode-01:6003 d56 2000.40 2789 -3.18 57 1 1 va-oldnode-01:6007 va-oldnode-01:6003 d57 2000.40 2789 -3.18 58 1 1 va-oldnode-01:6008 va-oldnode-01:6003 d58 2000.40 2789 -3.18 59 1 1 va-oldnode-01:6009 va-oldnode-01:6003 d59 2000.40 2789 -3.18 60 1 1 va-oldnode-01:6010 va-oldnode-01:6003 d60 2000.40 2789 -3.18 61 1 1 va-oldnode-01:6011 va-oldnode-01:6003 d61 2000.40 2789 -3.18 62 1 1 va-oldnode-01:6012 va-oldnode-01:6003 d62 2000.40 2789 -3.18 63 1 1 va-oldnode-01:6013 va-oldnode-01:6003 d63 2000.40 2789 -3.18 64 1 1 va-oldnode-01:6014 va-oldnode-01:6003 d64 2000.40 2789 -3.18 65 1 1 va-oldnode-01:6015 va-oldnode-01:6003 d65 2000.40 2788 -3.21 44 1 1 va-oldnode-02:6000 va-oldnode-02:6003 d44 269.43 376 -3.09 45 1 1 va-oldnode-02:6006 va-oldnode-02:6003 d45 269.43 376 -3.09 46 1 1 va-oldnode-02:6007 va-oldnode-02:6003 d46 269.43 376 -3.09 47 1 1 va-oldnode-02:6008 va-oldnode-02:6003 d47 269.43 376 -3.09 48 1 1 va-oldnode-02:6009 va-oldnode-02:6003 d48 269.43 376 -3.09 49 1 1 va-oldnode-02:6010 va-oldnode-02:6003 d49 269.43 376 -3.09 50 1 1 va-oldnode-02:6011 va-oldnode-02:6003 d50 269.43 376 -3.09 51 1 1 va-oldnode-02:6012 va-oldnode-02:6003 d51 269.43 375 -3.35 52 1 1 va-oldnode-02:6013 va-oldnode-02:6003 d52 269.43 375 -3.35 53 1 1 va-oldnode-02:6014 va-oldnode-02:6003 d53 269.43 375 -3.35 54 1 1 va-oldnode-02:6015 va-oldnode-02:6003 d54 269.43 375 -3.35 0 1 1 va-oldnode-03:6000 va-oldnode-03:6003 d0 2000.40 2789 -3.18 1 1 1 va-oldnode-03:6006 va-oldnode-03:6003 d1 2000.40 2789 -3.18 10 1 1 va-oldnode-03:6015 va-oldnode-03:6003 d10 2000.40 2788 -3.21 2 1 1 va-oldnode-03:6007 va-oldnode-03:6003 d2 2000.40 2789 -3.18 3 1 1 va-oldnode-03:6008 va-oldnode-03:6003 d3 2000.40 2789 -3.18 4 1 1 va-oldnode-03:6009 va-oldnode-03:6003 d4 2000.40 2789 -3.18 5 1 1 va-oldnode-03:6010 va-oldnode-03:6003 d5 2000.40 2789 -3.18 6 1 1 va-oldnode-03:6011 va-oldnode-03:6003 d6 2000.40 2789 -3.18 7 1 1 va-oldnode-03:6012 va-oldnode-03:6003 d7 2000.40 2789 -3.18 8 1 1 va-oldnode-03:6013 va-oldnode-03:6003 d8 2000.40 2789 -3.18 9 1 1 va-oldnode-03:6014 va-oldnode-03:6003 d9 2000.40 2789 -3.18 11 2 2 ca-oldnode-01:6000 ca-oldnode-01:6003 d11 2000.40 2979 3.42 12 2 2 ca-oldnode-01:6006 ca-oldnode-01:6003 d12 2000.40 2979 3.42 13 2 2 ca-oldnode-01:6007 ca-oldnode-01:6003 d13 2000.40 2979 3.42 15 2 2 ca-oldnode-01:6008 ca-oldnode-01:6003 d15 2000.40 2979 3.42 16 2 2 ca-oldnode-01:6009 ca-oldnode-01:6003 d16 2000.40 2979 3.42 17 2 2 ca-oldnode-01:6010 ca-oldnode-01:6003 d17 2000.40 2978 3.38 18 2 2 ca-oldnode-01:6011 ca-oldnode-01:6003 d18 2000.40 2978 3.38 19 2 2 ca-oldnode-01:6012 ca-oldnode-01:6003 d19 2000.40 2978 3.38 20 2 2 ca-oldnode-01:6013 ca-oldnode-01:6003 d20 2000.40 2978 3.38 21 2 2 ca-oldnode-01:6014 ca-oldnode-01:6003 d21 2000.40 2978 3.38 77 2 2 ca-oldnode-01:6015 ca-oldnode-01:6003 d77 2000.40 2978 3.38 33 2 2 ca-oldnode-02:6000 ca-oldnode-02:6003 d33 2000.40 2979 3.42 34 2 2 ca-oldnode-02:6006 ca-oldnode-02:6003 d34 2000.40 2979 3.42 35 2 2 ca-oldnode-02:6007 ca-oldnode-02:6003 d35 2000.40 2979 3.42 36 2 2 ca-oldnode-02:6008 ca-oldnode-02:6003 d36 2000.40 2979 3.42 37 2 2 ca-oldnode-02:6009 ca-oldnode-02:6003 d37 2000.40 2978 3.38 38 2 2 ca-oldnode-02:6010 ca-oldnode-02:6003 d38 2000.40 2978 3.38 39 2 2 ca-oldnode-02:6011 ca-oldnode-02:6003 d39 2000.40 2978 3.38 40 2 2 ca-oldnode-02:6012 ca-oldnode-02:6003 d40 2000.40 2978 3.38 41 2 2 ca-oldnode-02:6013 ca-oldnode-02:6003 d41 2000.40 2978 3.38 42 2 2 ca-oldnode-02:6014 ca-oldnode-02:6003 d42 2000.40 2978 3.38 43 2 2 ca-oldnode-02:6015 ca-oldnode-02:6003 d43 2000.40 2978 3.38 162 2 2 ca-oldnode-03:6000 ca-oldnode-03:6003 d162 1.00 2 38.89 163 2 2 ca-oldnode-03:6006 ca-oldnode-03:6003 d163 1.00 2 38.89 164 2 2 ca-oldnode-03:6007 ca-oldnode-03:6003 d164 1.00 2 38.89 165 2 2 ca-oldnode-03:6008 ca-oldnode-03:6003 d165 1.00 2 38.89 166 2 2 ca-oldnode-03:6009 ca-oldnode-03:6003 d166 1.00 1 -30.56 167 2 2 ca-oldnode-03:6010 ca-oldnode-03:6003 d167 1.00 1 -30.56 168 2 2 ca-oldnode-03:6011 ca-oldnode-03:6003 d168 1.00 1 -30.56
Currently rebalance is running and we see handoff partitions for all the policies.