accidently replied directly instead of to the list:

On Mon, Nov 13, 2023 at 10:43 AM Muralikrishna Gutha <mgutha@liveperson.com> wrote:

Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed [..] the storage utilization in the Primary Zone rapidly increased

This seems reasonable - what would have been your expected behavior?

we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes

You may be effected by min_part_hours - once a replicate of a part moves you can't move another replica of that same part in subsequent rebalance immediately.

Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data.

It may be necessary to share more specifics from your cluster topology - the zone & server & device weights

At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process.

perhaps you can reduce min_part_hours, but there is some risk of unavailability until the rebalance engine catches up with ring placement.

Our objective is to efficiently distribute the data across nodes

That is typically the goal of the ring rebalance algorithm, balance is a primary driver - but there are some constraints such as: moving only one replica of a part per rebalance and maximizing dispersion.

min_part_hours is a property set on the builder; you can view it with the swift-ring-builder CLI command:

[root@cloud 1fe628ae-2c4b-4e29-a995-d62021a17bd8]# pwd
/opt/ss/builder_configs/1fe628ae-2c4b-4e29-a995-d62021a17bd8
[root@cloud 1fe628ae-2c4b-4e29-a995-d62021a17bd8]# swift-ring-builder object.builder | head
object.builder, build version 165, id 90011ec6baaf4973a317771d3da63976
1024 partitions, 3.000000 replicas, 1 regions, 3 zones, 32 devices, 10.42 balance, 11.65 dispersion
The minimum number of hours before a partition can be reassigned is 0 (0:00:00 remaining)
The overload factor is 10.00% (0.100000)
Ring file object.ring.gz not found, probably it hasn't been written yet
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
24 1 1 192.168.25.156:6000 192.168.28.175:6003 d24 4.19 87 -9.37
25 1 1 192.168.25.156:6006 192.168.28.175:6003 d25 4.19 87 -9.37
26 1 1 192.168.25.156:6007 192.168.28.175:6003 d26 4.19 87 -9.37
27 1 1 192.168.25.156:6008 192.168.28.175:6003 d27 4.19 86 -10.42

sometimes it's reasonable to set it to zero - esp if you're monitoring handoff parts during a rebalance and being careful about when you rebalance and push rings. If you have a legacy swiftstack support contact you can reach out for advice - we might be able to help!

Clay Gerrard