[Openstack] Remove storage from the rings and then re-add..
sam at swiftstack.com
Mon Oct 20 16:42:53 UTC 2014
On 10/19/14, 11:24 PM, Shyam Prasad N wrote:
> I've written a few scripts to make adding of storage to swift cluster
> remakerings: removes existing ring files and builds new empty rings.
> addtorings: adds new IP:DISK pair to existing rings.
> removefromrings: removes an IP:DISK pair from existing rings.
> I'm facing a problem in one scenario though...
> After adding a few IP:DISK pairs to the swift cluster, and the cluster
> is in use for sometime. Later, I decided to add more storage to the
> cluster. For some reason, if the addtorings for the new storage fails
> after partial completion (i.e. container and account rings updated; but
> object ring update failed). So now I did a removefromrings to rollback
> my new changes. The remove request succeeds and gives a message "Will be
> removed from rings on next rebalance". But rebalance fails because the
> last rebalance was within 1 hr of the last rebalance. And re-adding the
> same IP:DISK pair to rings also fails; says "Already a part of rings".
> Is there anyway to workaround this problem? i.e. re-add the IP:DISK pair
> without having to wait for min_part_hours.
First, swift-ring-builder keeps backups of the builder files in an
adjacent directory named "backups". Don't try to back out an un-pushed
change by performing the inverse change; just grab the backup file and
overwrite the modified builder files.
(Note: an "un-pushed" change is one where the admin runs
swift-ring-builder, but does not end up copying ring.gz files out to the
cluster. Those changes can safely be thrown away since the cluster
doesn't actually know about them. If a change makes it into ring.gz
files and they are subsequently copied out to the cluster nodes, then
it's better to undo a change by performing a new operation to cancel it
Second, that part about rebalance failing isn't quite right. That
message is a warning, not an error, and its purpose is to tell the
administrator that they're not done yet. swift-ring-builder exits with 0
on success, 1 on warning, and 2 on error. Your scripts should take that
More information about the Openstack