On Thu, 2020-01-23 at 12:34 +1300, Mark Kirkwood wrote:
Hi,
We are wanting to increase the number of disks in each of our storage nodes - from 4 to 12.
I'm wondering whether it is better to:
1/ Add 1st new disk (with a reduced weight)...increase the weight until full, then repeat for next disk etc
2/ Add 'em all with a (much i.e 1/8 of that in 1/ ) reduced weight...increase the weights until done
Thoughts?
regards
Mark
Hi Mark,
I'd go with option 2 -- the quicker you can get all of the new disks helping with load, the better. Gradual weight adjustments seem like a good idea; they should help keep your replication traffic reasonable. Note that as long as you're waiting a full replication cycle between rebalances, though, swift should only be moving a single replica at a time, even if you added the new devices at full weight.
Of course, tripling capacity like this (assuming that the new disks are the same size as the existing ones) tends to take a while. You should probably familiarize yourself with the emergency replication options and consider enabling some of them until your rings reflect the new topology; see
* https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample... * https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample... and * https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample...
These can be really useful to speed up rebalances, though swift's durability guarantees take a bit of a hit -- so turn them back off once you've had a cycle or two with the drives at full weight! If the existing drives are full or nearly so (which IME tends to be the case when there's a large capacity increase), those may be necessary to get the system back to a state where it can make good progress.
Good luck!
Tim