<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0);">
<div>Thanks for the clarification!</div>
<div><br>
</div>
<div>Best,</div>
<div>-Minglong</div>
<div><br>
</div>
<div>On 11/29/13, 4:26 PM, "Samuel Merritt" <<a href="mailto:sam@swiftstack.com">sam@swiftstack.com</a>> wrote:</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>On 11/27/13 9:24 PM, Shao, Minglong wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>Thanks for your reply!</div>
<div>I understand the three PUTs by the proxy server and how the replicator</div>
<div>works.</div>
<div><br>
</div>
<div>What I don’t understand is the update of the container DB. The update is</div>
<div>sent by individual object servers which don’t know whether a PUT (from</div>
<div>the client’s perspective) succeeds.</div>
<div>Consider the following scenario:</div>
<div><br>
</div>
<div> 1. Proxy server sends three requests to three object servers.</div>
<div> 2. One object server writes the object successfully, sends an update to</div>
<div> the container DB and an “OK” reply to the proxy server. But the</div>
<div> other two fail, so they send “failed” to the proxy server.</div>
<div> 3. The proxy server sends back “failed” to the client because it</div>
<div> doesn’t meet the quorum. But the container DB still gets the update</div>
<div> to insert an entry of this object.</div>
</blockquote>
<div><br>
</div>
<div>The word "failed" is a bit nebulous here, and I think that's the source </div>
<div>of the confusion.</div>
<div><br>
</div>
<div>The proxy server doesn't send "failed" to the client; it sends back an </div>
<div>HTTP response with status code 503. This sounds like nitpicking, but </div>
<div>it's not. A 503 certainly doesn't indicate success, but neither does it </div>
<div>indicate that the requested operation failed. It really doesn't tell the </div>
<div>client anything except that something went wrong somewhere; whether that </div>
<div>constitutes a failure or not depends on the details.</div>
<div><br>
</div>
<div>It's possible that the object servers finished the upload successfully, </div>
<div>but the final stage of the upload took too long, and the proxy server </div>
<div>timed out the requests and sent a 503 to the client. In this case, one </div>
<div>could have all 3 (or N, really) replicas of the object successfully </div>
<div>written to disk, but the client still got a 503. Is that a failure?</div>
<div><br>
</div>
<div>It's possible that one object server finished the upload successfully, </div>
<div>but the others did not. In this case, the client will get a 503 due to </div>
<div>lack of quorum. However, let's say that the one, lonely copy of the </div>
<div>object is replicated before its disk malfunctions, and so the cluster </div>
<div>ends up with the full 3 (or N) replicas. Is that a failure?</div>
<div><br>
</div>
<div>It's possible that (again) one object server finished the upload </div>
<div>successfully, but the others did not. In this case, the client will get </div>
<div>a 503 due to lack of quorum. However, let's say that the one, lonely </div>
<div>copy of the object dies a horrible death due to disk malfunction, and so </div>
<div>the container listing contains an object that will always 404. Is that a </div>
<div>failure?</div>
<div><br>
</div>
<div>It's possible that the client is uploading an object and sends the last </div>
<div>bytes of the request body to the proxy, and before those critical last </div>
<div>bytes are sent to the object servers, the proxy's internal NIC fails, </div>
<div>the object-server requests all time out, and the client replies with a </div>
<div>503. Is that a failure? (Yes, this one definitely is a failure.)</div>
<div><br>
</div>
<div>I hope this helps clarify that a 503 indicates neither upload success </div>
<div>nor upload failure, but simply that something went wrong somewhere, and </div>
<div>that the object may exist and have full durability, may exist with poor </div>
<div>durability, or may not exist at all.</div>
<div><br>
</div>
<div>_______________________________________________</div>
<div>Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></div>
<div>Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a></div>
<div>Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></div>
<div><br>
</div>
</blockquote>
</body>
</html>