<div dir="ltr"><div>Thanks for your suggestions!</div><div>Mark was right about automated provisioning tools. I am using Saltstack for it.</div><div><br></div><div>The problem I try to solve is the automatic deployment. Adding and removing new nodes is the problematic thing because we can't just add or remove a full node at once (it can cause storm of data movement). We need to add / remove weight around 50 GB each 1 hour. So, 2 TB disk will be added or removed in 40 hours. It is a problematic to run a crond job on one machine because this job can fail (if the machine die or something like this).</div><div><br></div><div>Right now the workaround which I have is the next:</div><div>Every hour salt masters run a job to provision new nodes.</div><div>The job consists of several steps but the main idea is the next:</div><div>1) Node tries to get a global lock to change rings (in my case I check the time of last lock in MariaDB Galera cluster. If the last lock was more then 1hour ago then I can get a lock if no then I can not get a lock).</div><div>2) If the lock was obtained then I download the latest version of rings from OpenStack Swift and get an information about all nodes from rings. If any node has a different wight from the wight which I have set in Saltstack (Pillar) then I add / remove 50 weight. If the node has 0 weight then I remove it from the cluster. If I have point new nodes in Saltstack which hasn't been added to the cluster yet then I add them with weigth 50. </div><div>3) Rebalance the ring if any changes have occured. Update the ring in OpenStack Swift.</div><div><br></div><div>Also every hour the new ring will be copied to any node if it has a different ring.</div><div><br></div><div>I don't know whether it is a normal workaround or a strange one but I haven't found any other solutions. Please share if you have any ideas.</div><div><br></div><div>Best regards,</div><div>Alexandr </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 23, 2017 at 4:37 AM, Mark Kirkwood <span dir="ltr"><<a href="mailto:mark.kirkwood@catalyst.net.nz" target="_blank">mark.kirkwood@catalyst.net.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yes, one set of identical ring files is definitely the best option! Also the recon md5 ring check will likely complain forever if these alternative methods (for avoiding copying) are tried.<br>
<br>
<br>
More interesting is why people want to avoid the copy? I'm guessing it is because this type of thing is hard to do in some of the automated provisioning tools (ahem..looking at you puppet):<br>
<br>
- add disk on storage node<br>
<br>
- tell ring generation/master node(s) about it<br>
<br>
- make new rings on the node(s) above if new disks detected<br>
<br>
- copy new rings from node(s) in step 2 (for every node in cluster)<br>
<br>
<br>
Now the puppet-swift module does tell you how to do this (via exported resources), but it is not that simple (and in fact the examples in the module have mistakes that result in non identical rings proxy nodes...). Other complexities that make this tricky are multiple regions/geo replication - if each has its own puppet server...you can't (easily) use exported resources at all.<br>
<br>
Keeping the rings in swift is quite a cool idea - but can't be used when bootstrapping the cluster initially (probably not that difficult to work around tho).<br>
<br>
<br>
We are currently<br>
<br>
- keeping boostrap ring in puppet<br>
- add new disks/nodes manually (with helper scripts)<br>
- keep 'master' rings on all proxies<br>
- run ring rsync servers on all proxies<br>
- storage nodes get rings from proxies (puppet or manually)<br>
<br>
We run the recon md5 ring check as part of standard monitoring to pick up any mistakes we might do add devices.<br>
<br>
cheers<br>
<br>
Mark<span class=""><br>
<br>
On 23/01/17 03:16, Vladimir Maliaev wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Anyway you will copy ring files to new node. Which node you are going to copy from?<br>
I think that to have only one set of ring files is helpful in troubleshooting if something goes wrong.<br>
<br></span><span class="">
On Jan 22, 2017 6:08 AM, "Alexandr Porunov" <<a href="mailto:alexandr.porunov@gmail.com" target="_blank">alexandr.porunov@gmail.com</a> <mailto:<a href="mailto:alexandr.porunov@gmail.com" target="_blank">alexandr.porunov@gmail<wbr>.com</a>>> wrote:<br>
<br>
Hello,<br>
<br>
I read different articles about how to add a new node and they all<br>
says this:<br>
1) Add a node to ring builder<br>
2) Rebalance ring<br>
3) Copy ring to each node.<br>
<br>
Is it possible to skip "copy" step by doing first and second step<br>
on each node? Will then all nodes have the same ring?<br>
I.e. I mean to do like this:<br>
1) Add a node to the ring builder (Execute this step on each node)<br>
2) Rebalance the ring (Execute this step on each node)<br>
<br>
Will the ring be the same or we still have to copy the ring from<br>
one node to all others?<br>
<br>
Best regards,<br>
Alexandr<br>
<br>
______________________________<wbr>_________________<br>
Mailing list:<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck</a>><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br></span>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openst<wbr>ack.org</a>><br>
Unsubscribe :<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck</a>><span class=""><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
</span></blockquote>
<br>
</blockquote></div><br></div>