<div>Thanks for your quick reply John ~</div><div><br></div>Seems that I lost something with umount disk test before .<div><br></div><div>I'll try it later<br>appreciate ~<br><br><div class="gmail_quote">2012/9/7 John Dickinson <span dir="ltr"><<a href="mailto:me@not.mn" target="_blank">me@not.mn</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">you can force a replicator to push to a handoff node by unmounting the drive one of the primary replicas is on.<br>
<span class="HOEnZb"><font color="#888888"><br>
--John<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On Sep 6, 2012, at 9:00 AM, Kuo Hugo <<a href="mailto:tonytkdk@gmail.com">tonytkdk@gmail.com</a>> wrote:<br>
<br>
> Hi folks , John and Chmouel ,<br>
><br>
> I did post a question about this long time ago. And my test result is match to Chmouel's answer.<br>
><br>
> <a href="https://answers.launchpad.net/swift/+question/191924" target="_blank">https://answers.launchpad.net/swift/+question/191924</a><br>
> "The object replicator will push an object to a handoff node if another primary node returns that the drive the object is supposed to go on is bad. We don't push to handoff nodes on general errors, otherwise things like network partitions or rebooting machines would cause storms of unneeded handoff traffic."<br>
><br>
> But I read something different from John (or just my misunderstanding) , so want to clarify it.<br>
><br>
> Assumption :<br>
> Storage Nodes : 5 (each for one zone)<br>
> Zones : 5<br>
> Replica : 3<br>
> Disks : 2*5 ( 1 disk/per node )<br>
><br>
> Account AUTH_test<br>
> Container Con_1<br>
> Object Obj1<br>
><br>
><br>
> Partition 3430<br>
> Hash 6b342ac122448ef16bf1655d652bfe1e<br>
><br>
> Server:Port Device <a href="http://192.168.1.101:36000" target="_blank">192.168.1.101:36000</a> DISK1<br>
> Server:Port Device <a href="http://192.168.1.102:36000" target="_blank">192.168.1.102:36000</a> DISK1<br>
> Server:Port Device <a href="http://192.168.1.103:36000" target="_blank">192.168.1.103:36000</a> DISK1<br>
> Server:Port Device <a href="http://192.168.1.104:36000" target="_blank">192.168.1.104:36000</a> DISK1 [Handoff]<br>
> Server:Port Device <a href="http://192.168.1.105:36000" target="_blank">192.168.1.105:36000</a> DISK1 [Handoff]<br>
><br>
><br>
> curl -I -XHEAD "<a href="http://192.168.1.101:36000/DISK1/3430/AUTH_test/Con_1/Obj1" target="_blank">http://192.168.1.101:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>"<br>
> curl -I -XHEAD "<a href="http://192.168.1.102:36000/DISK1/3430/AUTH_test/Con_1/Obj1" target="_blank">http://192.168.1.102:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>"<br>
> curl -I -XHEAD "<a href="http://192.168.1.103:36000/DISK1/3430/AUTH_test/Con_1/Obj1" target="_blank">http://192.168.1.103:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>"<br>
> curl -I -XHEAD "<a href="http://192.168.1.104:36000/DISK1/3430/AUTH_test/Con_1/Obj1" target="_blank">http://192.168.1.104:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>" # [Handoff]<br>
> curl -I -XHEAD "<a href="http://192.168.1.105:36000/DISK1/3430/AUTH_test/Con_1/Obj1" target="_blank">http://192.168.1.105:36000/DISK1/3430/AUTH_test/Con_1/Obj1</a>" # [Handoff]<br>
><br>
><br>
> ssh 192.168.1.101 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/"<br>
> ssh 192.168.1.102 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/"<br>
> ssh 192.168.1.103 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/"<br>
> ssh 192.168.1.104 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/" # [Handoff]<br>
> ssh 192.168.1.105 "ls -lah /srv/node/DISK1/objects/3430/e1e/6b342ac122448ef16bf1655d652bfe1e/" # [Handoff]<br>
><br>
> Case :<br>
> Obj1 is already been uploaded to 3 primary devices properly. What kind of fails on "<a href="http://192.168.1.101:3600" target="_blank">192.168.1.101:3600</a> DISK1" will trigger replicator push a copy to "<a href="http://192.168.1.104:36000" target="_blank">192.168.1.104:36000</a> DISK1 [handoff] " device ?<br>
><br>
> In my past test , the replicator does not push a copy to handoff node for an "existing" object. Whatever network fail / reboot machine / umount disk , I think these are general errors from Chmouel mentioned before. But I'm not that sure about the meaning of "replicator will push an object to a handoff node if another primary node returns that the drive the object is supposed to go on is bad" . How object-replicator to know that the drive the object is supposed to go on is bad (I think replicator will never know it. Should it work with object-auditor ?)<br>
><br>
> How to produce a fail to trigger replicator push object to handoff node ?<br>
><br>
> In my consideration , for replicator pushes an object to handoff node there's a condition is that primary device does not have the object , also can not push into the device(<a href="http://192.168.1.101:36000" target="_blank">192.168.1.101:36000</a> DISK1). It might be moved to quarantine due to the object-auditor found the object is broken.<br>
><br>
> So that even the disk(<a href="http://192.168.1.101:3600" target="_blank">192.168.1.101:3600</a> DISK1) is still mounted and the target partition 3430 does not have Obj1 . Another node's object-replicator try to push it's Obj1 to "<a href="http://192.168.1.101:36000" target="_blank">192.168.1.101:36000</a> DISK1" , but unluckily , the "<a href="http://192.168.1.101:36000" target="_blank">192.168.1.101:36000</a> DISK1" is bad. So the object-replicator will push object to "<a href="http://192.168.1.104:36000" target="_blank">192.168.1.104:36000</a> DISK1 [handoff] " now .<br>
><br>
> That's my inference , please feel free to correct it . I'm really confusing about to produce the kind of fails for replicator to push object to handoff node .<br>
> Any idea would be great .<br>
><br>
><br>
> Cheers<br>
> --<br>
> +Hugo Kuo+<br>
> <a href="mailto:tonytkdk@gmail.com">tonytkdk@gmail.com</a><br>
> <a href="tel:%2B886%20935004793" value="+886935004793">+886 935004793</a><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>+Hugo Kuo+</div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com<br></a></div><div><a href="mailto:tonytkdk@gmail.com" target="_blank">+</a>886 935004793</div>
<br>
</div>