Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi guys,

we running in a described problem on our storage nodes.
The object auditor process throws errors because the system has no
available ports.

Mar  9 13:05:37 swift2 object-replicator: Error syncing with node:
{'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':
'10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22',
'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call
last):#012  File
"/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in
update#012    '', headers=self.headers).getresponse()#012  File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
157, in http_connect#012    ipaddr, port, method, path, headers,
query_string, ssl)#012  File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
189, in http_connect_raw#012    conn.endheaders()#012  File
"/usr/lib/python2.7/httplib.py", line 954, in endheaders#012   
self._send_output(message_body)#012  File
"/usr/lib/python2.7/httplib.py", line 814, in _send_output#012   
self.send(msg)#012  File "/usr/lib/python2.7/httplib.py", line 776, in
send#012    self.connect()#012  File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
108, in connect#012    return HTTPConnection.connect(self)#012  File
"/usr/lib/python2.7/httplib.py", line 757, in connect#012   
self.timeout, self.source_address)#012  File
"/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in
create_connection#012    raise error, msg#012error: [Errno 99] EADDRNOTAVAIL


:~# netstat --inet | grep TIME_WAIT | wc -l
63038

This value of used ports is on all nodes nearly the same and fluctuates
extremely. So i tuned the kernel and network stack of the Linux kernel
but without success.


# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# disable syn cookies
net.ipv4.tcp_syncookies = 0

# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

net.ipv4.ip_local_port_range = 18000 65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1
net.netfilter.nf_conntrack_tcp_timeout_established=600
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=5


The object-server conf-file:
[object-replicator]
recon_enable = yes
concurrency = 2
run_pause = 60
reclaim_age = 259200
interval = 60

[object-updater]
concurrency = 4
recon_enable = yes
recon_cache_path = /var/cache/swift
slowdown = 0.1

[object-auditor]
bytes_per_second = 3000000
files_per_second = 10
concurrency = 4
recon_enable = yes
recon_cache_path = /var/cache/swift



Have anyone a hint for me ?


Greetings
Heiko

- -- 
anynines.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJU/ZtBAAoJELxFogM4ixOFS88IANZZBzzFnFymWXRyuAQGjVpz
X7Os9Y9Jn41EOph4HHS9eablTc14YX4YiB/JvKj1KKJAUOkVoPfB5oC154hQ5Goa
i3f1qSWg3qEqv/lo5EvtX++B92Ut/68OSUblie1XGkivs6ZIfzeByzJqDgwdS2kV
UEMzyEw9K4oNFkyURts8vH4NX4FgqKIoaPaQh6qOe27YKEdWw9NJn3NbRzWncwVJ
R181jaerubZo8gYOVO9zYLHoPFLSxVft7zC6M0fHK6SqDUosA8zjperlvWChx2ZD
UnL3LAEs1BCSxnJw876AvH9nxwFwkZwioQeVW5inTtqxmvZRn0RnsCY/qzv51Oc=
=2fqn
-----END PGP SIGNATURE-----