<html><body>
<p><font size="2" face="sans-serif">Hi Shrinand,</font><br>
<br>
<font size="2" face="sans-serif">In HTTP1.0, client is required to add an additional header to the request so that</font><br>
<font size="2" face="sans-serif">the server kept the connection reused by multiple HTTP requests.</font><br>
<font size="2" face="sans-serif"> </font><font size="2" face="sans-serif"><i>Connection: Keep-Alive</i></font><br>
<font size="2" face="sans-serif">In HTTP1.1, all connections are considered persistent unless declared otherwise.</font><br>
<font size="2" face="sans-serif">The keepalive message is not used separately. I believe Swift is using the HTTP 1.1</font><br>
<font size="2" face="sans-serif">protocol. However, the HTTP server has default time out settings.</font><br>
<br>
<font size="2" face="sans-serif">In Swift proxy server, the default connection timeout is controlled in config file as</font><br>
<font size="2" face="sans-serif">below. You can try to look deeper inside of code how it is used.</font><br>
<font size="2" face="Monospace"># node_timeout = 10</font><br>
<font size="2" face="Monospace"># client_timeout = 60</font><br>
<font size="2" face="Monospace"># conn_timeout = 0.5</font><br>
<font size="2" face="sans-serif"># post_quorum_timeout = 0.5</font><br>
<br>
<font size="2" face="sans-serif">I'm also developing a trace tool for Swift </font><a href="https://review.openstack.org/#/c/93677/"><font size="2" face="sans-serif">here</font></a><font size="2" face="sans-serif">. You can try it and see if it can provide</font><br>
<font size="2" face="sans-serif">you more insight of this question. After you enable it, you can record the info</font><br>
<font size="2" face="sans-serif">into the logs that interests you such as remote client ip and port in each request. </font><br>
<font size="2" face="sans-serif">(currently this info is not recorded by default. you need to add it in the code. Or you</font><br>
<font size="2" face="sans-serif">can tell me what you need so that I update this patch.) Aggregate those logs and see </font><br>
<font size="2" face="sans-serif">how many ports on each ip/node are used at specific time slots.</font><br>
<br>
<font size="2" face="sans-serif">This is what I know. Hope this information is useful for you. </font><br>
<br>
<font size="2" face="sans-serif">-Edward Zhang</font><br>
<br>
<br>
<img width="16" height="16" src="cid:1__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Shrinand Javadekar ---2014-07-10 ÉÏÎç 01:53:28---Shrinand Javadekar <shrinand@maginatics.com>"><font size="2" color="#424282" face="sans-serif">Shrinand Javadekar ---2014-07-10 ÉÏÎç 01:53:28---Shrinand Javadekar <shrinand@maginatics.com></font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:2__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com); background-repeat: no-repeat; " width="40%">
<ul style="padding-left: 72pt"><font size="1" face="sans-serif"><b>Shrinand Javadekar <shrinand@maginatics.com></b></font><font size="1" face="sans-serif"> </font>
<p><font size="1" face="sans-serif">2014-07-10 ÉÏÎç 01:37</font></ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">"openstack@lists.openstack.org" <openstack@lists.openstack.org></font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">Re: [Openstack] [Swift] Running out of ports or fds?</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=C7BBF782DF9E27BF8f9e8a93df938@cn.ibm.com" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<tt><font size="2">Any ideas folks?<br>
<br>
On Tue, Jul 8, 2014 at 4:26 PM, Shrinand Javadekar<br>
<shrinand@maginatics.com> wrote:<br>
> Hi,<br>
><br>
> I have a question about the http connections made between the various<br>
> swift server processes. Particularly between the swift proxy server<br>
> and the swift object server.<br>
><br>
> I see that these servers do not use a persistent http connection<br>
> between them. So every blob get/put/delete request will create a new<br>
> connection, use it and tear it down. In a highly concurrent<br>
> environment with thousands of such operations happening per second,<br>
> there could be two problems:<br>
><br>
> i) Time required for creating the new connections could hamper performance.<br>
> ii) After the requests are complete, several connections will be in<br>
> the TIME_WAIT state and it might be possible that the proxy server and<br>
> object server node will run out of ports or fds.<br>
><br>
> If the proxy and object servers are on the same machine, the problem<br>
> is exacerbated. I have one such instance and at one point saw ~30K<br>
> sockets in the TIME_WAIT state. Though, this would've included<br>
> connections with the account server and container server also.<br>
><br>
> Does this analysis make sense? If yes, are there ways to do something<br>
> about it (other than asking clients to slow down :P)?<br>
><br>
> Thanks in advance.<br>
> -Shri<br>
<br>
_______________________________________________<br>
Mailing list: </font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></font></tt><tt><font size="2"><br>
Post to : openstack@lists.openstack.org<br>
Unsubscribe : </font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></font></tt><tt><font size="2"><br>
<br>
</font></tt><br>
</body></html>