<div dir="ltr">Hey Edward,<div><br></div><div>Thanks for the explanation. I had a discussion regarding this on the openstack-swift irc channel yesterday. Sam Merrit mentioned pretty much the points that you mention here.</div>
<div><br></div><div>To confirm this, I actually bumped up the client_chunk_size and object_chunk_size in the proxy-server.conf as well as disk_chunk_size and network_chunk_size in the object-server.conf and ran the same experiment. As expected these did not change the results. The client which is farther still results in slower PUT times compared to the closer one.</div>
<div><br></div><div>Also, I know that a single node swift cluster can have multiple replicas. I meant to say that my setup has a single node Swift instance with a single replica.</div><div><br></div><div>Just to summarize:</div>
<div><br></div><div>* In my setup, the object server is writing the data to SSDs. So the IO cost of writing to the storage is less.</div><div>* The proxy-server and object-server are co-located. So communication between them is fast.</div>
<div>* Therefore the dominating factor affecting the time taken by PUT requests is the network latency between client and swift proxy-server.</div><div>* Optimizing this would require tweaking either the TCP options on the client and server machines or changing the socket options appropriately in the application code.</div>
<div><br></div><div><div class="gmail_extra">-Shri<br><br><div class="gmail_quote">On Tue, Jun 24, 2014 at 8:17 PM, Hua ZZ Zhang <span dir="ltr"><<a href="mailto:zhuadl@cn.ibm.com" target="_blank">zhuadl@cn.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<p><font face="sans-serif">I don't think so. Since you are doing PUT object request, it will have network I/O operations b/w client/proxy server/object server</font><br>
<font face="sans-serif">and disk I/O operations on object server to save the data. The latency of sending data from client to proxy server should be affected</font><br>
<font face="sans-serif">by their network latency. So the object server might wait a very little more time to get those data transmitted from proxy server. </font><br>
<font face="sans-serif">Plus object server need to disk I/O to save object data. Ultimately the network latency can be ignored due to disk operation become </font><br>
<font face="sans-serif">the new bottleneck if your network latency improve a lot.</font><br>
<br>
<font face="sans-serif">Single node Swift cluster don't mean you could have only one replica. It can be more by configuring your install script. You should check it in your </font><br>
<font face="sans-serif">Swift Rings. Even you changed the network and disk chuck size, it is only used by Swift object server as buffer size to read and write data from </font><br>
<font face="sans-serif">network or disk. It does not change the default TCP/IP packet size which means you can't send the object data in one packet. The network latency</font><br>
<font face="sans-serif">will have impact on your object PUT request.</font><br>
<br>
<font face="sans-serif">-Edward</font><br>
<br>
<br>
<img width="16" height="16" src="cid:2__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Shrinand Javadekar ---2014-06-25 上午 01:14:07---Shrinand Javadekar <shrinand@maginatics.com>"><font color="#424282" face="sans-serif">Shrinand Javadekar ---2014-06-25 上午 01:14:07---Shrinand Javadekar <<a href="mailto:shrinand@maginatics.com" target="_blank">shrinand@maginatics.com</a>></font><br>
<br>
</p><table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td style="background-image:url(cid:3__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com);background-repeat:no-repeat" width="40%">
<ul style="padding-left:72pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <<a href="mailto:shrinand@maginatics.com" target="_blank">shrinand@maginatics.com</a>></b></font><font size="1" face="sans-serif"> </font>
<p><font size="1" face="sans-serif">2014-06-25 上午 01:14</font></p></ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">Hua ZZ Zhang/China/IBM@IBMCN</font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">"<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>" <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>></font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left:7pt"><font size="1" face="sans-serif">Re: [Openstack] [Swift] PUT requests sensitive to latency?</font></ul>
</td></tr>
</tbody></table>
<table border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="58"><img width="1" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" border="0" alt=""></td>
</tr>
</tbody></table>
</td></tr>
</tbody></table><div><div class="h5">
<br>
<font size="3" face="serif">Communication between proxy and object servers shouldn't be affected by the latency between the proxy server and the client, right? Also, I'm using a single node Swift cluster. So there should be only 1 copy of the object (along with any other I/Os required for the container and accound DBs).</font><br>
<br>
<font size="3" face="serif">Everything that happens on the Swift side should be the same (if there is no back-n-forth between the Swift server and client) irrespective of how much time it takes for communication between the Swift cluster and the client.</font><br>
<br>
<font size="3" face="serif">I had made one mistake when experimenting with the network_chunk_size and disk_chunk_size config options. These are supposed to go to the object-server.conf and not the proxy-server.conf. I made that change and restarted all the swift servers. However, I don't see any improvement. My current object-server.conf looks like this: </font><a href="http://pastie.org/private/exjiho1cbl80mbruythama" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://pastie.org/private/exjiho1cbl80mbruythama</u></font></a><br>
<br>
<font size="3" face="serif">What do you think?</font><br>
<br>
<font size="3" face="serif">-Shri</font><br>
<font size="3" face="serif"><br>
</font><br>
<font size="3" face="serif">On Tue, Jun 24, 2014 at 12:29 AM, Hua ZZ Zhang <</font><a href="mailto:zhuadl@cn.ibm.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>zhuadl@cn.ibm.com</u></font></a><font size="3" face="serif">> wrote:</font>
<ul style="padding-left:9pt"><font size="3" face="sans-serif">My guess is the object data need to be transmitted to Swift cluster before the status code returned.<br>
It can't be returned immediately before 2/3 I/O completed. Otherwise it is not consistent to tell client<br>
it succeed.</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
-Edward Zhang</font><font size="3" face="serif"><br>
<br>
<br>
</font><img src="cid:2__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="16" height="16" alt="Inactive hide details for Shrinand Javadekar ---2014-06-24 下午 03:12:14---Shrinand Javadekar <shrinand@maginatics.com>"><font size="3" color="#424282" face="sans-serif">Shrinand Javadekar ---2014-06-24 下午 03:12:14---Shrinand Javadekar <</font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>shrinand@maginatics.com</u></font></a><font size="3" color="#424282" face="sans-serif">></font><font size="3" face="serif"><br>
</font></ul>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="42%">
<ul style="padding-left:36pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <</b></font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><b><u>shrinand@maginatics.com</u></b></font></a><font size="1" face="sans-serif"><b>></b></font><font size="1" face="sans-serif"> </font><font size="3" face="serif"> </font>
<p><font size="1" face="sans-serif">2014-06-24 下午 03:05</font></p></ul>
</td><td width="58%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="12%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="88%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left:36pt"><font size="1" face="sans-serif">"</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">" <</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">></font></ul>
</td></tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="88%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td>
</tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="88%"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left:36pt"><font size="1" face="sans-serif">[Openstack] [Swift] PUT requests sensitive to latency?</font></ul>
</td></tr>
</tbody></table>
<table border="0" cellspacing="0" cellpadding="0">
<tbody><tr valign="top"><td width="58"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td><td width="336"><img src="cid:4__=C7BBF791DF9E1E428f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td>
</tr>
</tbody></table>
</td></tr>
</tbody></table>
<ul style="padding-left:9pt"><br>
<tt><font size="3">Hi,<br>
<br>
I have a single node swift cluster. I measured the time taken to<br>
complete a PUT request that originated from three different client<br>
machines. Each client was writing a single 256K byte object.<br>
<br>
Note that the time measured was only the time taken on the Swift<br>
cluster itself. I started the timer after the request was received by<br>
the swift proxy-server process and stopped it when that method<br>
returned an HTTP status to the client. This is not the time on the<br>
client side and therefore *ideally* should not be affected by the<br>
latency between the client and the Swift cluster.<br>
<br>
However, it appears that the above is not true. The time required to<br>
complete the request is related to the latency between the client and<br>
swift cluster.<br>
<br>
Here are the results:<br>
<br>
* Client 1:<br>
Ping time 28ms<br>
PUT request time: ~180 ms<br>
<br>
* Client 2:<br>
Ping time 4 ms<br>
PUT request time: ~35 ms<br>
<br>
* Client 3:<br>
Ping time 0.04 ms<br>
PUT request time: ~10 ms<br>
<br>
Details about the experiment:<br>
<br>
* This is a single node Swift installation (not devstack) and uses<br>
SSDs to store metadata as well as data. This is just a test setup. In<br>
production, we won't have SSDs for storing data.<br>
* The above numbers are average of 50 PUT requests.<br>
* The Swift cluster was not being used for anything else during the experiment.<br>
* The client used was the jclouds library written in java. I had<br>
disable a config option that used the Expect 100-Continue header; i.e.<br>
the requests were not using the Expect 100-Continue header.<br>
* I tried increasing the size of the following options in the<br>
proxy-server.conf and restarting Swift.<br>
disk_chunk_size = 262144<br>
network_chunk_size = 262144<br>
...<br>
[app:proxy-server]<br>
object_chunk_size = 262144<br>
client_chunk_size = 262144<br>
<br>
However, this didn't show any improvement in the time required for PUT requests.<br>
<br>
Am I missing anything? Does Swift require an extra round trip from the<br>
client for completing PUT requests? Any ways of avoiding that?<br>
<br>
Thanks in advance.<br>
-Shri<br>
</font></tt><br>
<tt><font size="3">_______________________________________________<br>
Mailing list: </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><tt><font size="3"><br>
Post to : </font></tt><a href="mailto:openstack@lists.openstack.org" target="_blank"><tt><font size="3" color="#0000FF"><u>openstack@lists.openstack.org</u></font></tt></a><tt><font size="3"><br>
Unsubscribe : </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><tt><font size="3"><br>
</font></tt><font size="3" face="serif"><br>
</font></ul>
<br>
</div></div><i>(See attached file: pic09941.gif)</i>
<p></p><p></p></div></blockquote></div><br></div></div></div>