<div dir="ltr">I thought those tracebacks only showed up with old versions of eventlet or and eventlet_debug = true?<div><br></div><div>In my experience that normally indicates a client disconnect on a chucked encoding transfer request (request w/o a content-length). Do you know if your clients are using transfer encoding chunked?</div>
<div><br></div><div>Are you seeing the 408 make it's way out to the client? It wasn't clear to me if you only see these tracebacks on the object-servers or in the proxy logs as well? Perhaps only one of the three disks involved in the PUT are timing out and the client still gets a successful response?</div>
<div><br></div><div>As the disks fill up replication and auditing is going to consume more disk resources - you may have to tune the concurrency and rate settings on those daemons. If the errors happen consistently you could try running with background consistency processes temporarily disabled and rule out if they're causing disk contention on your setup with your config.</div>
<div><br></div><div>-Clay</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, May 9, 2014 at 8:54 AM, Ben Nemec <span dir="ltr"><<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This is a development list, and your question sounds more usage-related. Please ask your question on the users list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
<br>
Thanks.<br>
<br>
-Ben<div><div class="h5"><br>
<br>
On 05/09/2014 06:57 AM, Shyam Prasad N wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi,<br>
<br>
I have a two node swift cluster receiving continuous traffic (mostly<br>
overwrites for existing objects) of 1GB files each.<br>
<br>
Soon after the traffic started, I'm seeing the following traceback from<br>
some transactions...<br>
Traceback (most recent call last):<br>
File "/home/eightkpc/swift/swift/<u></u>proxy/controllers/obj.py", line 692,<br>
in PUT<br>
chunk = next(data_source)<br>
File "/home/eightkpc/swift/swift/<u></u>proxy/controllers/obj.py", line 559,<br>
in <lambda><br>
data_source = iter(lambda: reader(self.app.client_chunk_<u></u>size), '')<br>
File "/home/eightkpc/swift/swift/<u></u>common/utils.py", line 2362, in read<br>
chunk = self.wsgi_input.read(*args, **kwargs)<br>
File "/usr/lib/python2.7/dist-<u></u>packages/eventlet/wsgi.py", line 147,<br>
in read<br>
return self._chunked_read(self.rfile, length)<br>
File "/usr/lib/python2.7/dist-<u></u>packages/eventlet/wsgi.py", line 137,<br>
in _chunked_read<br>
self.chunk_length = int(rfile.readline().split(";"<u></u>, 1)[0], 16)<br>
ValueError: invalid literal for int() with base 16: '' (txn:<br>
tx14e2df7680fd472fb92f0-<u></u>00536ca4f0) (client_ip: 10.3.0.101)<br>
<br>
Seeing the following errors on storage logs...<br>
object-server: 10.3.0.102 - - [09/May/2014:01:36:49 +0000] "PUT<br>
/xvdg/492/AUTH_test/8kpc/<u></u>30303A30323A30333A30343A30353A<u></u>30396AEF6B53000000007B000000.<u></u>2.data"<br>
408 - "PUT<br>
<a href="http://10.3.0.102:8080/v1/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data" target="_blank">http://10.3.0.102:8080/v1/<u></u>AUTH_test/8kpc/<u></u>30303A30323A30333A30343A30353A<u></u>30396AEF6B53000000007B000000.<u></u>2.data</a>"<br>
"txf3b4e5f677004474bbd2f-<u></u>00536c30d1" "proxy-server 12241" 95.6405 "-"<br>
<br>
It's success sometimes, but mostly 408 errors. I don't see any other<br>
logs for the transaction ID. or around these 408 errors in the log<br>
files. Is this a disk timeout issue? These are only 1GB files and normal<br>
writes to files on these disks are quite fast.<br>
<br>
The timeouts from the swift proxy files are...<br>
root@bulkstore-112:~# grep -R timeout /etc/swift/*<br>
/etc/swift/proxy-server.conf:<u></u>client_timeout = 600<br>
/etc/swift/proxy-server.conf:<u></u>node_timeout = 600<br>
/etc/swift/proxy-server.conf:<u></u>recoverable_node_timeout = 600<br>
<br>
Can someone help me troubleshoot this issue?<br>
<br>
--<br>
-Shyam<br>
<br>
<br></div></div>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote></div><br></div>