<html><body>
<p><font size="2" face="sans-serif">Shrinand,</font><br>
<br>
<font size="2" face="sans-serif">I have done some tracing work here and you maybe interested to try.</font><br>
<font size="2" face="sans-serif">This new patch can help you to record timing of each hop of request if you enable it in log options.</font><br>
<font size="2" face="sans-serif">This might give you more insights and help you to diagnose network latency problem.</font><br>
<a href="https://review.openstack.org/#/c/93677/"><font size="2" face="sans-serif">https://review.openstack.org/#/c/93677/</font></a><br>
<font size="2" face="sans-serif">Any feedback is welcomed!</font><br>
<br>
<font size="2" face="sans-serif">-Edward Zhang</font><br>
<br>
<br>
<img width="16" height="16" src="cid:2__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Shrinand Javadekar ---2014-06-26 下午 01:57:11---Shrinand Javadekar <shrinand@maginatics.com>"><font size="2" color="#424282" face="sans-serif">Shrinand Javadekar ---2014-06-26 下午 01:57:11---Shrinand Javadekar <shrinand@maginatics.com></font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:3__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com); background-repeat: no-repeat; " width="40%">
<ul style="padding-left: 72pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <shrinand@maginatics.com></b></font><font size="1" face="sans-serif"> </font>
<p><font size="1" face="sans-serif">2014-06-26 下午 01:57</font></ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">Hua ZZ Zhang/China/IBM@IBMCN</font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">"openstack@lists.openstack.org" <openstack@lists.openstack.org></font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">Re: [Openstack] [Swift] PUT requests sensitive to latency?</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<font size="3" face="serif">Hey Edward,</font><br>
<br>
<font size="3" face="serif">Thanks for the explanation. I had a discussion regarding this on the openstack-swift irc channel yesterday. Sam Merrit mentioned pretty much the points that you mention here.</font><br>
<br>
<font size="3" face="serif">To confirm this, I actually bumped up the client_chunk_size and object_chunk_size in the proxy-server.conf as well as disk_chunk_size and network_chunk_size in the object-server.conf and ran the same experiment. As expected these did not change the results. The client which is farther still results in slower PUT times compared to the closer one.</font><br>
<br>
<font size="3" face="serif">Also, I know that a single node swift cluster can have multiple replicas. I meant to say that my setup has a single node Swift instance with a single replica.</font><br>
<br>
<font size="3" face="serif">Just to summarize:</font><br>
<br>
<font size="3" face="serif">* In my setup, the object server is writing the data to SSDs. So the IO cost of writing to the storage is less.</font><br>
<font size="3" face="serif">* The proxy-server and object-server are co-located. So communication between them is fast.</font><br>
<font size="3" face="serif">* Therefore the dominating factor affecting the time taken by PUT requests is the network latency between client and swift proxy-server.</font><br>
<font size="3" face="serif">* Optimizing this would require tweaking either the TCP options on the client and server machines or changing the socket options appropriately in the application code.</font><br>
<br>
<font size="3" face="serif">-Shri<br>
</font><br>
<font size="3" face="serif">On Tue, Jun 24, 2014 at 8:17 PM, Hua ZZ Zhang <</font><a href="mailto:zhuadl@cn.ibm.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>zhuadl@cn.ibm.com</u></font></a><font size="3" face="serif">> wrote:</font>
<ul style="padding-left: 9pt"><font size="3" face="sans-serif">I don't think so. Since you are doing PUT object request, it will have network I/O operations b/w client/proxy server/object server<br>
and disk I/O operations on object server to save the data. The latency of sending data from client to proxy server should be affected<br>
by their network latency. So the object server might wait a very little more time to get those data transmitted from proxy server. <br>
Plus object server need to disk I/O to save object data. Ultimately the network latency can be ignored due to disk operation become <br>
the new bottleneck if your network latency improve a lot.</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
Single node Swift cluster don't mean you could have only one replica. It can be more by configuring your install script. You should check it in your <br>
Swift Rings. Even you changed the network and disk chuck size, it is only used by Swift object server as buffer size to read and write data from <br>
network or disk. It does not change the default TCP/IP packet size which means you can't send the object data in one packet. The network latency<br>
will have impact on your object PUT request.</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
-Edward</font><font size="3" face="serif"><br>
<br>
<br>
</font><img src="cid:2__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="16" height="16" alt="Inactive hide details for Shrinand Javadekar ---2014-06-25 上午 01:14:07---Shrinand Javadekar <shrinand@maginatics.com>"><font size="3" color="#424282" face="sans-serif">Shrinand Javadekar ---2014-06-25 上午 01:14:07---Shrinand Javadekar <</font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>shrinand@maginatics.com</u></font></a><font size="3" color="#424282" face="sans-serif">></font><font size="3" face="serif"><br>
</font></ul>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="42%">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <</b></font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><b><u>shrinand@maginatics.com</u></b></font></a><font size="1" face="sans-serif"><b>></b></font><font size="1" face="sans-serif"> </font><font size="3" face="serif"> </font>
<p><font size="1" face="sans-serif">2014-06-25 上午 01:14</font></ul>
</td><td width="58%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif">Hua ZZ Zhang/China/IBM@IBMCN</font></ul>
</td></tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif">"</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">" <</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">></font></ul>
</td></tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif">Re: [Openstack] [Swift] PUT requests sensitive to latency?</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td><td width="336"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td></tr>
</table>
</td></tr>
</table>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
Communication between proxy and object servers shouldn't be affected by the latency between the proxy server and the client, right? Also, I'm using a single node Swift cluster. So there should be only 1 copy of the object (along with any other I/Os required for the container and accound DBs).<br>
<br>
Everything that happens on the Swift side should be the same (if there is no back-n-forth between the Swift server and client) irrespective of how much time it takes for communication between the Swift cluster and the client.<br>
<br>
I had made one mistake when experimenting with the network_chunk_size and disk_chunk_size config options. These are supposed to go to the object-server.conf and not the proxy-server.conf. I made that change and restarted all the swift servers. However, I don't see any improvement. My current object-server.conf looks like this: </font><a href="http://pastie.org/private/exjiho1cbl80mbruythama" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://pastie.org/private/exjiho1cbl80mbruythama</u></font></a><font size="3" face="serif"><br>
<br>
What do you think?<br>
<br>
-Shri<br>
<br>
<br>
On Tue, Jun 24, 2014 at 12:29 AM, Hua ZZ Zhang <</font><a href="mailto:zhuadl@cn.ibm.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>zhuadl@cn.ibm.com</u></font></a><font size="3" face="serif">> wrote: </font>
<ul style="padding-left: 36pt"><font size="3" face="sans-serif">My guess is the object data need to be transmitted to Swift cluster before the status code returned.<br>
It can't be returned immediately before 2/3 I/O completed. Otherwise it is not consistent to tell client<br>
it succeed.<br>
<br>
-Edward Zhang</font><font size="3" face="serif"><br>
<br>
<br>
</font><img src="cid:2__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="16" height="16" alt="Inactive hide details for Shrinand Javadekar ---2014-06-24 下午 03:12:14---Shrinand Javadekar <shrinand@maginatics.com>"><font size="3" color="#424282" face="sans-serif">Shrinand Javadekar ---2014-06-24 下午 03:12:14---Shrinand Javadekar <</font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>shrinand@maginatics.com</u></font></a><font size="3" color="#424282" face="sans-serif">></font></ul>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="42%">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <</b></font><a href="mailto:shrinand@maginatics.com" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><b><u>shrinand@maginatics.com</u></b></font></a><font size="1" face="sans-serif"><b>></b></font><font size="1" face="sans-serif"> </font><font size="3" face="serif"> </font>
<p><font size="1" face="sans-serif">2014-06-24 下午 03:05</font></ul>
</td><td width="58%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif">"</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">" <</font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" face="sans-serif">></font></ul>
</td></tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td></tr>
<tr valign="top"><td width="12%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="58" height="1"><div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="88%"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1">
<ul style="padding-left: 36pt"><font size="1" face="sans-serif">[Openstack] [Swift] PUT requests sensitive to latency?</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td><td width="336"><img src="cid:4__=C7BBF790DFB01C9A8f9e8a93df938@cn.ibm.com" width="1" height="1" align="bottom"></td></tr>
</table>
</td></tr>
</table>
<ul style="padding-left: 36pt"><tt><font size="3"><br>
Hi,<br>
<br>
I have a single node swift cluster. I measured the time taken to<br>
complete a PUT request that originated from three different client<br>
machines. Each client was writing a single 256K byte object.<br>
<br>
Note that the time measured was only the time taken on the Swift<br>
cluster itself. I started the timer after the request was received by<br>
the swift proxy-server process and stopped it when that method<br>
returned an HTTP status to the client. This is not the time on the<br>
client side and therefore *ideally* should not be affected by the<br>
latency between the client and the Swift cluster.<br>
<br>
However, it appears that the above is not true. The time required to<br>
complete the request is related to the latency between the client and<br>
swift cluster.<br>
<br>
Here are the results:<br>
<br>
* Client 1:<br>
Ping time 28ms<br>
PUT request time: ~180 ms<br>
<br>
* Client 2:<br>
Ping time 4 ms<br>
PUT request time: ~35 ms<br>
<br>
* Client 3:<br>
Ping time 0.04 ms<br>
PUT request time: ~10 ms<br>
<br>
Details about the experiment:<br>
<br>
* This is a single node Swift installation (not devstack) and uses<br>
SSDs to store metadata as well as data. This is just a test setup. In<br>
production, we won't have SSDs for storing data.<br>
* The above numbers are average of 50 PUT requests.<br>
* The Swift cluster was not being used for anything else during the experiment.<br>
* The client used was the jclouds library written in java. I had<br>
disable a config option that used the Expect 100-Continue header; i.e.<br>
the requests were not using the Expect 100-Continue header.<br>
* I tried increasing the size of the following options in the<br>
proxy-server.conf and restarting Swift.<br>
disk_chunk_size = 262144<br>
network_chunk_size = 262144<br>
...<br>
[app:proxy-server]<br>
object_chunk_size = 262144<br>
client_chunk_size = 262144<br>
<br>
However, this didn't show any improvement in the time required for PUT requests.<br>
<br>
Am I missing anything? Does Swift require an extra round trip from the<br>
client for completing PUT requests? Any ways of avoiding that?<br>
<br>
Thanks in advance.<br>
-Shri</font></tt><font size="3" face="serif"><br>
</font><tt><font size="3"><br>
_______________________________________________<br>
Mailing list: </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><tt><font size="3"><br>
Post to : </font></tt><a href="mailto:openstack@lists.openstack.org" target="_blank"><tt><font size="3" color="#0000FF"><u>openstack@lists.openstack.org</u></font></tt></a><tt><font size="3"><br>
Unsubscribe : </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><font size="3" face="serif"><br>
</font></ul>
<br>
<font size="3" face="serif"><i>(See attached file: pic09941.gif)</i></font><font size="3" face="serif"> </font></ul>
<br>
<i>(See attached file: pic08688.gif)</i>
<p></body></html>