Open Stack

On 25/01/20 10:23 am, Tim Burke wrote:

> On Thu, 2020-01-23 at 12:34 +1300, Mark Kirkwood wrote:
>> Hi,
>>
>> We are wanting to increase the number of disks in each of our
>> storage
>> nodes - from 4 to 12.
>>
>> I'm wondering whether it is better to:
>>
>> 1/ Add 1st new disk (with a reduced weight)...increase the weight
>> until
>> full, then repeat for next disk etc
>>
>> 2/ Add 'em all with a (much i.e 1/8 of that in 1/ ) reduced
>> weight...increase the weights until done
>>
>> Thoughts?
>>
>> regards
>>
>> Mark
>>
>>
> Hi Mark,
>
> I'd go with option 2 -- the quicker you can get all of the new disks
> helping with load, the better. Gradual weight adjustments seem like a
> good idea; they should help keep your replication traffic reasonable.
> Note that as long as you're waiting a full replication cycle between
> rebalances, though, swift should only be moving a single replica at a
> time, even if you added the new devices at full weight.
>
> Of course, tripling capacity like this (assuming that the new disks are
> the same size as the existing ones) tends to take a while. You should
> probably familiarize yourself with the emergency replication options
> and consider enabling some of them until your rings reflect the new
> topology; see
>
>   *
> https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample#L290-L298
>   *
> https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample#L300-L307
> and
>   *
> https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample#L353-L364
>
> These can be really useful to speed up rebalances, though swift's
> durability guarantees take a bit of a hit -- so turn them back off once
> you've had a cycle or two with the drives at full weight! If the
> existing drives are full or nearly so (which IME tends to be the case
> when there's a large capacity increase), those may be necessary to get
> the system back to a state where it can make good progress.
>
Thanks Tim!

What I ended I doing (and is still in progress) is:

- initially adding a single disk at 50%, and waiting until a replication 
cycle had completed (pretty much just assessing impact), then I quickly 
changed to

- adding 3 disks (1 per region) at 16.66% each as this worked much better

- as more disks were added I was able to increase the addition % (50% 
now) as the bottleneck seemed to be read rate from the source (existing) 
disks

The disks are enterprise level SATA, plugged into a raid controller with 
a battery cache. We are not using the raid facility, only its battery to 
accelerate writes. So far I'm not maxing out the write ability on the 
target (new) disks. So I'm hoping to keep increasing the addition % to 
the point where I can add 3 disks @ 100% each and still have a 
replication cycle complete before I add the next lot!

regards

Mark