On 25/01/20 10:23 am, Tim Burke wrote:
On Thu, 2020-01-23 at 12:34 +1300, Mark Kirkwood wrote:
Hi,
We are wanting to increase the number of disks in each of our storage nodes - from 4 to 12.
I'm wondering whether it is better to:
1/ Add 1st new disk (with a reduced weight)...increase the weight until full, then repeat for next disk etc
2/ Add 'em all with a (much i.e 1/8 of that in 1/ ) reduced weight...increase the weights until done
Thoughts?
regards
Mark
Hi Mark,
I'd go with option 2 -- the quicker you can get all of the new disks helping with load, the better. Gradual weight adjustments seem like a good idea; they should help keep your replication traffic reasonable. Note that as long as you're waiting a full replication cycle between rebalances, though, swift should only be moving a single replica at a time, even if you added the new devices at full weight.
Of course, tripling capacity like this (assuming that the new disks are the same size as the existing ones) tends to take a while. You should probably familiarize yourself with the emergency replication options and consider enabling some of them until your rings reflect the new topology; see
https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample...
https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample... and
https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample...
These can be really useful to speed up rebalances, though swift's durability guarantees take a bit of a hit -- so turn them back off once you've had a cycle or two with the drives at full weight! If the existing drives are full or nearly so (which IME tends to be the case when there's a large capacity increase), those may be necessary to get the system back to a state where it can make good progress.
Thanks Tim!
What I ended I doing (and is still in progress) is:
- initially adding a single disk at 50%, and waiting until a replication cycle had completed (pretty much just assessing impact), then I quickly changed to
- adding 3 disks (1 per region) at 16.66% each as this worked much better
- as more disks were added I was able to increase the addition % (50% now) as the bottleneck seemed to be read rate from the source (existing) disks
The disks are enterprise level SATA, plugged into a raid controller with a battery cache. We are not using the raid facility, only its battery to accelerate writes. So far I'm not maxing out the write ability on the target (new) disks. So I'm hoping to keep increasing the addition % to the point where I can add 3 disks @ 100% each and still have a replication cycle complete before I add the next lot!
regards
Mark