Open Stack

Hello Klaus,

sorry for my late repsonse.


root at swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6000 | wc -l
40819
root at swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6001 | wc -l
15136


Yeah all on the same host. I notice ~30 GET/s, so not much.


@Clay, no i don't notice massiv client disconnects.


Thanks
Heiko

Am 10.03.2015 um 10:39 schrieb Klaus Schürmann:
> Hello Heiko,
>
> what kind of connections do you see in TIME_WAIT status?
> All connections to the object-server (Port 6000) ?
> Are the container- and account-server on the same host?
>
> How many PUTs and GETs do you receive per second?
>
> Best Regards
> Klaus
>
> -----Ursprüngliche Nachricht-----
> Von: Heiko Krämer [mailto:hkraemer at anynines.com]
> Gesendet: Montag, 9. März 2015 14:08
> An: openstack at lists.openstack.org
> Betreff: [Openstack] Swift and TIME_WAIT network stack problem
>
>
> Hi guys,
>
> we running in a described problem on our storage nodes.
> The object auditor process throws errors because the system has no
> available ports.
>
> Mar  9 13:05:37 swift2 object-replicator: Error syncing with node:
> {'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':
> '10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22',
> 'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call
> last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in
> update#012    '', headers=self.headers).getresponse()#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 157, in http_connect#012    ipaddr, port, method, path, headers,
> query_string, ssl)#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 189, in http_connect_raw#012    conn.endheaders()#012  File
> "/usr/lib/python2.7/httplib.py", line 954, in endheaders#012   
> self._send_output(message_body)#012  File
> "/usr/lib/python2.7/httplib.py", line 814, in _send_output#012   
> self.send(msg)#012  File "/usr/lib/python2.7/httplib.py", line 776, in
> send#012    self.connect()#012  File
> "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
> 108, in connect#012    return HTTPConnection.connect(self)#012  File
> "/usr/lib/python2.7/httplib.py", line 757, in connect#012   
> self.timeout, self.source_address)#012  File
> "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in
> create_connection#012    raise error, msg#012error: [Errno 99]
> EADDRNOTAVAIL
>
>
> :~# netstat --inet | grep TIME_WAIT | wc -l
> 63038
>
> This value of used ports is on all nodes nearly the same and
> fluctuates extremely. So i tuned the kernel and network stack of the
> Linux kernel but without success.
>
>
> # disable TIME_WAIT.. wait..
> net.ipv4.tcp_tw_recycle=1
> net.ipv4.tcp_tw_reuse=1
>
> # disable syn cookies
> net.ipv4.tcp_syncookies = 0
>
> # double amount of allowed conntrack
> net.ipv4.netfilter.ip_conntrack_max = 262144
>
> net.ipv4.ip_local_port_range = 18000 65535
> net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1
> net.netfilter.nf_conntrack_tcp_timeout_established=600
> net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
> net.ipv4.tcp_fin_timeout=15
> net.ipv4.tcp_keepalive_intvl=30
> net.ipv4.tcp_keepalive_probes=5
>
>
> The object-server conf-file:
> [object-replicator]
> recon_enable = yes
> concurrency = 2
> run_pause = 60
> reclaim_age = 259200
> interval = 60
>
> [object-updater]
> concurrency = 4
> recon_enable = yes
> recon_cache_path = /var/cache/swift
> slowdown = 0.1
>
> [object-auditor]
> bytes_per_second = 3000000
> files_per_second = 10
> concurrency = 4
> recon_enable = yes
> recon_cache_path = /var/cache/swift
>
>
>
> Have anyone a hint for me ?
>
>
> Greetings
> Heiko
>
> --
> anynines.com
>
>
>
> _______________________________________________
> Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-- 
anynines.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150327/6e532e4a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150327/6e532e4a/attachment.sig>