<div><font color="#3333ff" size="4">Hi folks , John and Chmouel ,</font></div><div><br></div><div>I did post a question about this long time ago. And my test result is match to Chmouel's answer. </div><div><br></div><div>
<u><a href="https://answers.launchpad.net/swift/+question/191924">https://answers.launchpad.net/swift/+question/191924</a></u></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px;background-color:rgb(255,255,255)">"The object replicator will push an object to a handoff node if another primary node returns that the drive the object is supposed to go on is bad. We don't push to handoff nodes on general errors, otherwise things like network partitions or rebooting machines would cause storms of unneeded handoff traffic."</span></div>
</blockquote><div><br></div><div>But I read something different from John (or just my misunderstanding) , so want to clarify it.</div><div><br></div><div><font color="#3333ff" size="4">Assumption : </font></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div>Storage Nodes : 5 (each for one zone)</div><div>Zones : 5</div><div>Replica : 3</div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>Disks : 2*5 ( 1 disk/per node )</div></blockquote>
<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Account AUTH_test</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Container Con_1</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Object Obj1</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><br></font></b></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><br></font></b></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Partition 3430</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Hash 6b342ac122448ef16bf1655d652bfe1e</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><br></font></b></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Server:Port Device <a href="http://192.168.1.101:36000">192.168.1.101:36000</a> DISK1</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Server:Port Device <a href="http://192.168.1.102:36000">192.168.1.102:36000</a> DISK1</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Server:Port Device <a href="http://192.168.1.103:36000">192.168.1.103:36000</a> DISK1 </span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><b id="internal-source-marker_0.2753794798627496" style="font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Server:Port Device <a href="http://192.168.1.104:36000">192.168.1.104:36000</a> DISK1 [Handoff]</span></font></b></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><b id="internal-source-marker_0.2753794798627496" style="font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Server:Port Device <a href="http://192.168.1.105:36000">192.168.1.105:36000</a> DISK1 [Handoff]<br>
<br></span></font></b></b></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">curl -I -XHEAD "<a href="http://192.168.1.101:36000/DISK1/3430/AUTH_test/Con_1/Obj1">http://192.168.1.101:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>"</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">curl -I -XHEAD "<a href="http://192.168.1.102:36000/DISK1/3430/AUTH_test/Con_1/Obj1">http://192.168.1.102:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>"</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">curl -I -XHEAD "<a href="http://192.168.1.103:36000/DISK1/3430/AUTH_test/Con_1/Obj1">http://192.168.1.103:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>" </span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><b id="internal-source-marker_0.2753794798627496" style="font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">curl -I -XHEAD "<a href="http://192.168.1.104:36000/DISK1/3430/AUTH_test/Con_1/Obj1">http://192.168.1.104:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>" # [Handoff]</span></font></b></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><b id="internal-source-marker_0.2753794798627496" style="font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">curl -I -XHEAD "<a href="http://192.168.1.105:36000/DISK1/3430/AUTH_test/Con_1/Obj1">http://192.168.1.105:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>" # [Handoff]<br>
<br></span></font></b></b></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ssh 192.168.1.101 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/"</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ssh 192.168.1.102 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/"</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ssh 192.168.1.103 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/" </span></font></b></blockquote>
</blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ssh 192.168.1.104 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/" # [Handoff]</span></font></b></blockquote>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b id="internal-source-marker_0.2753794798627496" style="font-family:'LiHei Pro';font-weight:normal"><b id="internal-source-marker_0.2753794798627496" style="font-weight:normal"><font size="1"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ssh 192.168.1.105 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/" # [Handoff]</span></font></b></b></blockquote>
</blockquote><div><br></div><div><font size="4">Case :</font> </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>Obj1 is already been uploaded to 3 primary devices properly. What kind of fails on <font color="#ff0000">"<b><a href="http://192.168.1.101:3600">192.168.1.101:3600</a> DISK1"</b></font> will trigger replicator push a copy to <font color="#3333ff">"<b><a href="http://192.168.1.104:36000">192.168.1.104:36000</a> DISK1 [handoff]</b> "</font> device ?</div>
</blockquote><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><br></div><div>In my past test , the replicator does not push a copy to handoff node for an "existing" object. Whatever network fail / reboot machine / umount disk , I think these are general errors from Chmouel mentioned before. But I'm not that sure about the meaning of "<span style="background-color:rgb(255,255,255);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><font color="#3333ff">replicator will push an object to a handoff node if another primary node returns that the drive the object is supposed to go on is bad</font></span><span style="background-color:rgb(255,255,255);color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px">" . How object-replicator to know that the drive the object is supposed to go on is bad (I think replicator will never know it. Should it work with object-auditor ?) </span></div>
<div><span style="background-color:rgb(255,255,255);color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><br></span></div></blockquote>
<span style="background-color:rgb(255,255,255);color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;line-height:18px"><font size="4">How to produce a fail to trigger replicator push object to handoff node ?</font></span></div>
<div><font color="#333333" face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><span style="font-size:12px;line-height:18px"><br></span></font><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><span style="background-color:rgb(255,255,255)"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><span style="font-size:12px;line-height:18px"><font color="#333333">In my consideration , for replicator pushes an object to handoff node there's a condition is that primary device does not have the object , also can not push into the device</font><font color="#ff0000">(</font></span></font></span><span style="background-color:rgb(255,255,255);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><font color="#ff0000"><a href="http://192.168.1.101:36000">192.168.1.101:36000</a> DISK1)</font></span><span style="font-size:12px;line-height:18px;color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;background-color:rgb(255,255,255)">. It might be moved to </span><span style="font-size:12px;line-height:18px;color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;background-color:rgb(255,255,255)">quarantine</span><span style="font-size:12px;line-height:18px;color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;background-color:rgb(255,255,255)"> due to the object-auditor found the object is broken. </span></div>
<div><span style="font-size:12px;line-height:18px;color:rgb(51,51,51);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;background-color:rgb(255,255,255)"><br></span></div><div><span style="background-color:rgb(255,255,255)"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><span style="font-size:12px;line-height:18px"><font color="#333333">So that even the disk(</font><font color="#cc0000"><a href="http://192.168.1.101:3600">192.168.1.101:3600</a> DISK1</font><font color="#333333">) is still mounted and the target partition 3430 does not have Obj1 . Another node's object-replicator try to push it's Obj1 to "</font></span></font></span><span style="background-color:rgb(255,255,255);color:rgb(204,0,0);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><a href="http://192.168.1.101:36000">192.168.1.101:36000</a> DISK1" </span><span style="background-color:rgb(255,255,255)"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><span style="font-size:12px;line-height:18px">, but unluckily , the "</span></font></span><span style="background-color:rgb(255,255,255);color:rgb(204,0,0);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><a href="http://192.168.1.101:36000">192.168.1.101:36000</a> DISK1</span><span style="background-color:rgb(255,255,255);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px">" is bad. So the object-replicator will push object to </span><font color="#3333ff">"<b><a href="http://192.168.1.104:36000">192.168.1.104:36000</a> DISK1 [handoff]</b> " </font>now .</div>
</blockquote></div><div><span style="background-color:rgb(255,255,255);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"><br></span></div><div><span style="background-color:rgb(255,255,255);font-size:12px;line-height:18px"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif">That's my inference , please feel free to correct it . I'm really confusing about to produce the kind of fails for replicator to push object to handoff node . </font></span></div>
<div><span style="background-color:rgb(255,255,255);font-size:12px;line-height:18px"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif">Any idea would be great .</font></span></div><div><span style="background-color:rgb(255,255,255);font-size:12px;line-height:18px"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><br>
</font></span></div><div><span style="background-color:rgb(255,255,255);font-size:12px;line-height:18px"><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><br></font></span></div><div><font face="Ubuntu, Bitstream Vera Sans, DejaVu Sans, Tahoma, sans-serif"><span style="font-size:12px;line-height:18px">Cheers</span></font><span style="background-color:rgb(255,255,255);font-family:Ubuntu,'Bitstream Vera Sans','DejaVu Sans',Tahoma,sans-serif;font-size:12px;line-height:18px"> </span></div>
-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div><br>