<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello Klaus,<br>
<br>
sorry for my late repsonse.<br>
<br>
<br>
root@swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6000 | wc -l<br>
40819<br>
root@swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6001 | wc -l<br>
15136<br>
<br>
<br>
Yeah all on the same host. I notice ~30 GET/s, so not much.<br>
<br>
<br>
@Clay, no i don't notice massiv client disconnects.<br>
<br>
<br>
Thanks <br>
Heiko<br>
<br>
Am 10.03.2015 um 10:39 schrieb Klaus Schürmann:<br>
<span style="white-space: pre;">> Hello Heiko,<br>
><br>
> what kind of connections do you see in TIME_WAIT status?<br>
> All connections to the object-server (Port 6000) ?<br>
> Are the container- and account-server on the same host?<br>
><br>
> How many PUTs and GETs do you receive per second?<br>
><br>
> Best Regards<br>
> Klaus<br>
><br>
> -----Ursprüngliche Nachricht-----<br>
> Von: Heiko Krämer [<a class="moz-txt-link-freetext" href="mailto:hkraemer@anynines.com">mailto:hkraemer@anynines.com</a>] <br>
> Gesendet: Montag, 9. März 2015 14:08<br>
> An: <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> Betreff: [Openstack] Swift and TIME_WAIT network stack
problem<br>
><br>
></span><br>
<blockquote type="cite">Hi guys,<br>
<br>
we running in a described problem on our storage nodes.<br>
The object auditor process throws errors because the system has no
available ports.<br>
<br>
Mar 9 13:05:37 swift2 object-replicator: Error syncing with node:<br>
{'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':<br>
'10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip':
'10.0.0.22',<br>
'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most
recent call<br>
last):#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line
282, in<br>
update#012 '', headers=self.headers).getresponse()#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py",
line<br>
157, in http_connect#012 ipaddr, port, method, path, headers,<br>
query_string, ssl)#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py",
line<br>
189, in http_connect_raw#012 conn.endheaders()#012 File<br>
"/usr/lib/python2.7/httplib.py", line 954, in endheaders#012 <br>
self._send_output(message_body)#012 File<br>
"/usr/lib/python2.7/httplib.py", line 814, in _send_output#012 <br>
self.send(msg)#012 File "/usr/lib/python2.7/httplib.py", line
776, in<br>
send#012 self.connect()#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py",
line<br>
108, in connect#012 return HTTPConnection.connect(self)#012
File<br>
"/usr/lib/python2.7/httplib.py", line 757, in connect#012 <br>
self.timeout, self.source_address)#012 File
"/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line
59, in<br>
create_connection#012 raise error, msg#012error: [Errno 99]
EADDRNOTAVAIL<br>
<br>
<br>
:~# netstat --inet | grep TIME_WAIT | wc -l<br>
63038<br>
<br>
This value of used ports is on all nodes nearly the same and
fluctuates extremely. So i tuned the kernel and network stack of
the Linux kernel but without success.<br>
<br>
<br>
# disable TIME_WAIT.. wait..<br>
net.ipv4.tcp_tw_recycle=1<br>
net.ipv4.tcp_tw_reuse=1<br>
<br>
# disable syn cookies<br>
net.ipv4.tcp_syncookies = 0<br>
<br>
# double amount of allowed conntrack<br>
net.ipv4.netfilter.ip_conntrack_max = 262144<br>
<br>
net.ipv4.ip_local_port_range = 18000 65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1<br>
net.netfilter.nf_conntrack_tcp_timeout_established=600<br>
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30<br>
net.ipv4.tcp_fin_timeout=15<br>
net.ipv4.tcp_keepalive_intvl=30<br>
net.ipv4.tcp_keepalive_probes=5<br>
<br>
<br>
The object-server conf-file:<br>
[object-replicator]<br>
recon_enable = yes<br>
concurrency = 2<br>
run_pause = 60<br>
reclaim_age = 259200<br>
interval = 60<br>
<br>
[object-updater]<br>
concurrency = 4<br>
recon_enable = yes<br>
recon_cache_path = /var/cache/swift<br>
slowdown = 0.1<br>
<br>
[object-auditor]<br>
bytes_per_second = 3000000<br>
files_per_second = 10<br>
concurrency = 4<br>
recon_enable = yes<br>
recon_cache_path = /var/cache/swift<br>
<br>
<br>
<br>
Have anyone a hint for me ?<br>
<br>
<br>
Greetings<br>
Heiko<br>
<br>
--<br>
anynines.com<br>
</blockquote>
<span style="white-space: pre;">><br>
><br>
><br>
> _______________________________________________<br>
> Mailing list:
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> Post to : <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> Unsubscribe :
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></span><br>
<br>
-- <br>
anynines.com<br>
<br>
</body>
</html>