<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Kun Huang</b> <span dir="ltr"><<a href="mailto:aragongareth@gmail.com">aragongareth@gmail.com</a>></span><br>
Date: 2013/2/27<br>Subject: [Openstack] [swift] some code understanding<br>To: openstack <<a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>><br><br><br><div dir="ltr">Hi swift developer,<div>
<br></div><div>I'm confused about implementation of ring structure.</div><div><br></div><div>in the RingBuilder, line 671 ~ 681</div><div><br></div><div>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold"> for</span> <span style="margin:0px;padding:0px;border:0px">part</span><span style="margin:0px;padding:0px;border:0px">,</span> <span style="margin:0px;padding:0px;border:0px">replace_replicas</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px">reassign_parts</span><span style="margin:0px;padding:0px;border:0px">:</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,136);font-style:italic"># Gather up what other tiers (zones, ip_ports, and devices) the</span></div><div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,136);font-style:italic"># replicas not-to-be-moved are in for this part.</span></div><div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">other_replicas</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">defaultdict</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">int</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">unique_tiers_by_tier_len</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">defaultdict</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">set</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">for</span> <span style="margin:0px;padding:0px;border:0px">replica</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">xrange</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,153)">self</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">replicas</span><span style="margin:0px;padding:0px;border:0px">):</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">if</span> <span style="margin:0px;padding:0px;border:0px">replica</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">not</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px">replace_replicas</span><span style="margin:0px;padding:0px;border:0px">:</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">dev</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,153)">self</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">devs</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px;color:rgb(153,153,153)">self</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">_replica2part2dev</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">replica</span><span style="margin:0px;padding:0px;border:0px">][</span><span style="margin:0px;padding:0px;border:0px">part</span><span style="margin:0px;padding:0px;border:0px">]]</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">for</span> <span style="margin:0px;padding:0px;border:0px">tier</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px">tiers_for_dev</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">dev</span><span style="margin:0px;padding:0px;border:0px">):</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">other_replicas</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">tier</span><span style="margin:0px;padding:0px;border:0px">]</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">+=</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">1</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">unique_tiers_by_tier_len</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">len</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">tier</span><span style="margin:0px;padding:0px;border:0px">)]</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">add</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">tier</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="color:rgb(51,51,51);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="font-size:12.222222328186035px;line-height:17.77777862548828px;margin:0px;padding:0px;border:0px"><br></span></div><br>this while loop results<font color="#000000"> in</font><font color="#0000ff"> other_replicas</font> and <font color="#0000ff">unique_tiers_by_tier_len, </font>but I don't have a confirmed understanding about these two data.<br>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"><span style><font color="#0000ff"><br></font></span></div>and in line 684 to 725: (I removed comments)</pre>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px"> tier</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">()</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">depth</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">1</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">while</span> <span style="margin:0px;padding:0px;border:0px">depth</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold"><=</span> <span style="margin:0px;padding:0px;border:0px">max_tier_depth</span><span style="margin:0px;padding:0px;border:0px">:</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">candidate_tiers</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">tier2children</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">tier</span><span style="margin:0px;padding:0px;border:0px">]</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">candidates_with_replicas</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> \</div><div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">unique_tiers_by_tier_len</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">len</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">tier</span><span style="margin:0px;padding:0px;border:0px">)</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">+</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">1</span><span style="margin:0px;padding:0px;border:0px">]</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">if</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">len</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">candidate_tiers</span><span style="margin:0px;padding:0px;border:0px">)</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">></span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">len</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">candidates_with_replicas</span><span style="margin:0px;padding:0px;border:0px">):</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">for</span> <span style="margin:0px;padding:0px;border:0px">t</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">reversed</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">candidate_tiers</span><span style="margin:0px;padding:0px;border:0px">):</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">if</span> <span style="margin:0px;padding:0px;border:0px">other_replicas</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">t</span><span style="margin:0px;padding:0px;border:0px">]</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">==</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">0</span><span style="margin:0px;padding:0px;border:0px">:</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">tier</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">t</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">break</span></div><div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">else</span><span style="margin:0px;padding:0px;border:0px">:</span></div><div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">min_count</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">min</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">other_replicas</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">t</span><span style="margin:0px;padding:0px;border:0px">]</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">for</span> <span style="margin:0px;padding:0px;border:0px">t</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px">candidate_tiers</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">tier</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">t</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">for</span> <span style="margin:0px;padding:0px;border:0px">t</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">in</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,134,179)">reversed</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">candidate_tiers</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;font-weight:bold">if</span> <span style="margin:0px;padding:0px;border:0px">other_replicas</span><span style="margin:0px;padding:0px;border:0px">[</span><span style="margin:0px;padding:0px;border:0px">t</span><span style="margin:0px;padding:0px;border:0px">]</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">==</span> <span style="margin:0px;padding:0px;border:0px">min_count</span><span style="margin:0px;padding:0px;border:0px">)</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">next</span><span style="margin:0px;padding:0px;border:0px">()</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px">depth</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">+=</span> <span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">1</span></div>
<div style="font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;color:rgb(51,51,51);margin:0px;padding:0px 0px 0px 10px;border:0px">
<span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)"><br></span></div>this loop search the tier tree, from () to (zone, ip, device), for finding a tier, and choosing a dev by that tier, and storing dev id in part key(line 762, that's a kind of final ring structure?). And the propose of tree structure is:</pre>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px">"""</pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><span style="font-family:arial,sans-serif;font-size:13px;white-space:normal">This is already supported in Swift with the concept of availability zones. Swift will place each replica in different availability zones, if possible. If you only have one zone, Swift will place the replicas on different machines. If you only have one machine, Swift will place the replicas on different drives.</span><br>
</pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><span style="font-family:arial">-- John</span><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px">""", right?</pre>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px">And the last loop line 732 to line 760 only works on sorting for next big loop? (line 683)</pre>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px">The propose of rebalance is to put every replica on correct partition, and put every partition on correct device? </pre>
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px">
Gareth</pre><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px"><br></pre></pre></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></div><br>