Open Stack

On 10/19/14, 11:24 PM, Shyam Prasad N wrote:
> Hi,
>
> I've written a few scripts to make adding of storage to swift cluster
> easier:
> remakerings: removes existing ring files and builds new empty rings.
> addtorings: adds new IP:DISK pair to existing rings.
> removefromrings: removes an IP:DISK pair from existing rings.
>
> I'm facing a problem in one scenario though...
> After adding a few IP:DISK pairs to the swift cluster, and the cluster
> is in use for sometime. Later, I decided to add more storage to the
> cluster. For some reason, if the addtorings for the new storage fails
> after partial completion (i.e. container and account rings updated; but
> object ring update failed).  So now I did a removefromrings to rollback
> my new changes. The remove request succeeds and gives a message "Will be
> removed from rings on next rebalance". But rebalance fails because the
> last rebalance was within 1 hr of the last rebalance. And re-adding the
> same IP:DISK pair to rings also fails; says "Already a part of rings".
>
> Is there anyway to workaround this problem? i.e. re-add the IP:DISK pair
> without having to wait for min_part_hours.

Two things:

First, swift-ring-builder keeps backups of the builder files in an 
adjacent directory named "backups". Don't try to back out an un-pushed 
change by performing the inverse change; just grab the backup file and 
overwrite the modified builder files.

(Note: an "un-pushed" change is one where the admin runs 
swift-ring-builder, but does not end up copying ring.gz files out to the 
cluster. Those changes can safely be thrown away since the cluster 
doesn't actually know about them. If a change makes it into ring.gz 
files and they are subsequently copied out to the cluster nodes, then 
it's better to undo a change by performing a new operation to cancel it 
out.)

Second, that part about rebalance failing isn't quite right. That 
message is a warning, not an error, and its purpose is to tell the 
administrator that they're not done yet. swift-ring-builder exits with 0 
on success, 1 on warning, and 2 on error. Your scripts should take that 
into account.