[Openstack] [Swift] How to add a new node without ring copy?
alexandr.porunov at gmail.com
Mon Jan 23 07:29:13 UTC 2017
Thanks for your suggestions!
Mark was right about automated provisioning tools. I am using Saltstack for
The problem I try to solve is the automatic deployment. Adding and removing
new nodes is the problematic thing because we can't just add or remove a
full node at once (it can cause storm of data movement). We need to add /
remove weight around 50 GB each 1 hour. So, 2 TB disk will be added or
removed in 40 hours. It is a problematic to run a crond job on one machine
because this job can fail (if the machine die or something like this).
Right now the workaround which I have is the next:
Every hour salt masters run a job to provision new nodes.
The job consists of several steps but the main idea is the next:
1) Node tries to get a global lock to change rings (in my case I check the
time of last lock in MariaDB Galera cluster. If the last lock was more then
1hour ago then I can get a lock if no then I can not get a lock).
2) If the lock was obtained then I download the latest version of rings
from OpenStack Swift and get an information about all nodes from rings. If
any node has a different wight from the wight which I have set in Saltstack
(Pillar) then I add / remove 50 weight. If the node has 0 weight then I
remove it from the cluster. If I have point new nodes in Saltstack which
hasn't been added to the cluster yet then I add them with weigth 50.
3) Rebalance the ring if any changes have occured. Update the ring in
Also every hour the new ring will be copied to any node if it has a
I don't know whether it is a normal workaround or a strange one but I
haven't found any other solutions. Please share if you have any ideas.
On Mon, Jan 23, 2017 at 4:37 AM, Mark Kirkwood <
mark.kirkwood at catalyst.net.nz> wrote:
> Yes, one set of identical ring files is definitely the best option! Also
> the recon md5 ring check will likely complain forever if these alternative
> methods (for avoiding copying) are tried.
> More interesting is why people want to avoid the copy? I'm guessing it is
> because this type of thing is hard to do in some of the automated
> provisioning tools (ahem..looking at you puppet):
> - add disk on storage node
> - tell ring generation/master node(s) about it
> - make new rings on the node(s) above if new disks detected
> - copy new rings from node(s) in step 2 (for every node in cluster)
> Now the puppet-swift module does tell you how to do this (via exported
> resources), but it is not that simple (and in fact the examples in the
> module have mistakes that result in non identical rings proxy nodes...).
> Other complexities that make this tricky are multiple regions/geo
> replication - if each has its own puppet server...you can't (easily) use
> exported resources at all.
> Keeping the rings in swift is quite a cool idea - but can't be used when
> bootstrapping the cluster initially (probably not that difficult to work
> around tho).
> We are currently
> - keeping boostrap ring in puppet
> - add new disks/nodes manually (with helper scripts)
> - keep 'master' rings on all proxies
> - run ring rsync servers on all proxies
> - storage nodes get rings from proxies (puppet or manually)
> We run the recon md5 ring check as part of standard monitoring to pick up
> any mistakes we might do add devices.
> On 23/01/17 03:16, Vladimir Maliaev wrote:
>> Anyway you will copy ring files to new node. Which node you are going to
>> copy from?
>> I think that to have only one set of ring files is helpful in
>> troubleshooting if something goes wrong.
>> On Jan 22, 2017 6:08 AM, "Alexandr Porunov" <alexandr.porunov at gmail.com
>> <mailto:alexandr.porunov at gmail.com>> wrote:
>> I read different articles about how to add a new node and they all
>> says this:
>> 1) Add a node to ring builder
>> 2) Rebalance ring
>> 3) Copy ring to each node.
>> Is it possible to skip "copy" step by doing first and second step
>> on each node? Will then all nodes have the same ring?
>> I.e. I mean to do like this:
>> 1) Add a node to the ring builder (Execute this step on each node)
>> 2) Rebalance the ring (Execute this step on each node)
>> Will the ring be the same or we still have to copy the ring from
>> one node to all others?
>> Best regards,
>> Mailing list:
>> Post to : openstack at lists.openstack.org
>> <mailto:openstack at lists.openstack.org>
>> Unsubscribe :
>> Mailing list: http://lists.openstack.org/cgi
>> Post to : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Openstack