Open Stack

Another possibility is a large number of client disconnects - which could
happen serving mostly media content for progressive download; or maybe lots
of X-Newest requests.

I've seen situations where the proxy could try harder to close down backend
connections - https://bugs.launchpad.net/swift/+bug/667956

-Clay

On Tue, Mar 10, 2015 at 2:39 AM, Klaus Schürmann <
klaus.schuermann at mediabeam.com> wrote:

> Hello Heiko,
>
> what kind of connections do you see in TIME_WAIT status?
> All connections to the object-server (Port 6000) ?
> Are the container- and account-server on the same host?
>
> How many PUTs and GETs do you receive per second?
>
> Best Regards
> Klaus
>
> -----Ursprüngliche Nachricht-----
> Von: Heiko Krämer [mailto:hkraemer at anynines.com]
> Gesendet: Montag, 9. März 2015 14:08
> An: openstack at lists.openstack.org
> Betreff: [Openstack] Swift and TIME_WAIT network stack problem
>
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi guys,
>
> we running in a described problem on our storage nodes.
> The object auditor process throws errors because the system has no
> available ports.
>
> Mar  9 13:05:37 swift2 object-replicator: Error syncing with node:
> {'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':
> '10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22',
> 'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call
> last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in
> update#012    '', headers=self.headers).getresponse()#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 157, in http_connect#012    ipaddr, port, method, path, headers,
> query_string, ssl)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 189, in http_connect_raw#012    conn.endheaders()#012  File
> "/usr/lib/python2.7/httplib.py", line 954, in endheaders#012
> self._send_output(message_body)#012  File
> "/usr/lib/python2.7/httplib.py", line 814, in _send_output#012
> self.send(msg)#012  File "/usr/lib/python2.7/httplib.py", line 776, in
> send#012    self.connect()#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 108, in connect#012    return HTTPConnection.connect(self)#012  File
> "/usr/lib/python2.7/httplib.py", line 757, in connect#012
> self.timeout, self.source_address)#012  File
> "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in
> create_connection#012    raise error, msg#012error: [Errno 99]
> EADDRNOTAVAIL
>
>
> :~# netstat --inet | grep TIME_WAIT | wc -l
> 63038
>
> This value of used ports is on all nodes nearly the same and fluctuates
> extremely. So i tuned the kernel and network stack of the Linux kernel but
> without success.
>
>
> # disable TIME_WAIT.. wait..
> net.ipv4.tcp_tw_recycle=1
> net.ipv4.tcp_tw_reuse=1
>
> # disable syn cookies
> net.ipv4.tcp_syncookies = 0
>
> # double amount of allowed conntrack
> net.ipv4.netfilter.ip_conntrack_max = 262144
>
> net.ipv4.ip_local_port_range = 18000 65535
> net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1
> net.netfilter.nf_conntrack_tcp_timeout_established=600
> net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
> net.ipv4.tcp_fin_timeout=15
> net.ipv4.tcp_keepalive_intvl=30
> net.ipv4.tcp_keepalive_probes=5
>
>
> The object-server conf-file:
> [object-replicator]
> recon_enable = yes
> concurrency = 2
> run_pause = 60
> reclaim_age = 259200
> interval = 60
>
> [object-updater]
> concurrency = 4
> recon_enable = yes
> recon_cache_path = /var/cache/swift
> slowdown = 0.1
>
> [object-auditor]
> bytes_per_second = 3000000
> files_per_second = 10
> concurrency = 4
> recon_enable = yes
> recon_cache_path = /var/cache/swift
>
>
>
> Have anyone a hint for me ?
>
>
> Greetings
> Heiko
>
> - --
> anynines.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJU/ZtBAAoJELxFogM4ixOFS88IANZZBzzFnFymWXRyuAQGjVpz
> X7Os9Y9Jn41EOph4HHS9eablTc14YX4YiB/JvKj1KKJAUOkVoPfB5oC154hQ5Goa
> i3f1qSWg3qEqv/lo5EvtX++B92Ut/68OSUblie1XGkivs6ZIfzeByzJqDgwdS2kV
> UEMzyEw9K4oNFkyURts8vH4NX4FgqKIoaPaQh6qOe27YKEdWw9NJn3NbRzWncwVJ
> R181jaerubZo8gYOVO9zYLHoPFLSxVft7zC6M0fHK6SqDUosA8zjperlvWChx2ZD
> UnL3LAEs1BCSxnJw876AvH9nxwFwkZwioQeVW5inTtqxmvZRn0RnsCY/qzv51Oc=
> =2fqn
> -----END PGP SIGNATURE-----
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150311/70ea6c45/attachment.html>