<div dir="ltr">Another possibility is a large number of client disconnects - which could happen serving mostly media content for progressive download; or maybe lots of X-Newest requests.<div><br></div><div>I've seen situations where the proxy could try harder to close down backend connections - <a href="https://bugs.launchpad.net/swift/+bug/667956">https://bugs.launchpad.net/swift/+bug/667956</a></div><div><br></div><div>-Clay</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 10, 2015 at 2:39 AM, Klaus Schürmann <span dir="ltr"><<a href="mailto:klaus.schuermann@mediabeam.com" target="_blank">klaus.schuermann@mediabeam.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Heiko,<br>
<br>
what kind of connections do you see in TIME_WAIT status?<br>
All connections to the object-server (Port 6000) ?<br>
Are the container- and account-server on the same host?<br>
<br>
How many PUTs and GETs do you receive per second?<br>
<br>
Best Regards<br>
Klaus<br>
<br>
-----Ursprüngliche Nachricht-----<br>
Von: Heiko Krämer [mailto:<a href="mailto:hkraemer@anynines.com">hkraemer@anynines.com</a>]<br>
Gesendet: Montag, 9. März 2015 14:08<br>
An: <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Betreff: [Openstack] Swift and TIME_WAIT network stack problem<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi guys,<br>
<br>
we running in a described problem on our storage nodes.<br>
The object auditor process throws errors because the system has no available ports.<br>
<br>
Mar 9 13:05:37 swift2 object-replicator: Error syncing with node:<br>
{'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':<br>
'10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22',<br>
'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call<br>
last):#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in<br>
update#012 '', headers=self.headers).getresponse()#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line<br>
157, in http_connect#012 ipaddr, port, method, path, headers,<br>
query_string, ssl)#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line<br>
189, in http_connect_raw#012 conn.endheaders()#012 File<br>
"/usr/lib/python2.7/httplib.py", line 954, in endheaders#012<br>
self._send_output(message_body)#012 File<br>
"/usr/lib/python2.7/httplib.py", line 814, in _send_output#012<br>
self.send(msg)#012 File "/usr/lib/python2.7/httplib.py", line 776, in<br>
send#012 self.connect()#012 File<br>
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line<br>
108, in connect#012 return HTTPConnection.connect(self)#012 File<br>
"/usr/lib/python2.7/httplib.py", line 757, in connect#012<br>
self.timeout, self.source_address)#012 File "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in<br>
create_connection#012 raise error, msg#012error: [Errno 99] EADDRNOTAVAIL<br>
<br>
<br>
:~# netstat --inet | grep TIME_WAIT | wc -l<br>
63038<br>
<br>
This value of used ports is on all nodes nearly the same and fluctuates extremely. So i tuned the kernel and network stack of the Linux kernel but without success.<br>
<br>
<br>
# disable TIME_WAIT.. wait..<br>
net.ipv4.tcp_tw_recycle=1<br>
net.ipv4.tcp_tw_reuse=1<br>
<br>
# disable syn cookies<br>
net.ipv4.tcp_syncookies = 0<br>
<br>
# double amount of allowed conntrack<br>
net.ipv4.netfilter.ip_conntrack_max = 262144<br>
<br>
net.ipv4.ip_local_port_range = 18000 65535 net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1<br>
net.netfilter.nf_conntrack_tcp_timeout_established=600<br>
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30<br>
net.ipv4.tcp_fin_timeout=15<br>
net.ipv4.tcp_keepalive_intvl=30<br>
net.ipv4.tcp_keepalive_probes=5<br>
<br>
<br>
The object-server conf-file:<br>
[object-replicator]<br>
recon_enable = yes<br>
concurrency = 2<br>
run_pause = 60<br>
reclaim_age = 259200<br>
interval = 60<br>
<br>
[object-updater]<br>
concurrency = 4<br>
recon_enable = yes<br>
recon_cache_path = /var/cache/swift<br>
slowdown = 0.1<br>
<br>
[object-auditor]<br>
bytes_per_second = 3000000<br>
files_per_second = 10<br>
concurrency = 4<br>
recon_enable = yes<br>
recon_cache_path = /var/cache/swift<br>
<br>
<br>
<br>
Have anyone a hint for me ?<br>
<br>
<br>
Greetings<br>
Heiko<br>
<br>
- --<br>
<a href="http://anynines.com" target="_blank">anynines.com</a><br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1<br>
<br>
iQEcBAEBAgAGBQJU/ZtBAAoJELxFogM4ixOFS88IANZZBzzFnFymWXRyuAQGjVpz<br>
X7Os9Y9Jn41EOph4HHS9eablTc14YX4YiB/JvKj1KKJAUOkVoPfB5oC154hQ5Goa<br>
i3f1qSWg3qEqv/lo5EvtX++B92Ut/68OSUblie1XGkivs6ZIfzeByzJqDgwdS2kV<br>
UEMzyEw9K4oNFkyURts8vH4NX4FgqKIoaPaQh6qOe27YKEdWw9NJn3NbRzWncwVJ<br>
R181jaerubZo8gYOVO9zYLHoPFLSxVft7zC6M0fHK6SqDUosA8zjperlvWChx2ZD<br>
UnL3LAEs1BCSxnJw876AvH9nxwFwkZwioQeVW5inTtqxmvZRn0RnsCY/qzv51Oc=<br>
=2fqn<br>
-----END PGP SIGNATURE-----<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</div></div></blockquote></div><br></div>