<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Clay, <br>
First of all, thanks for the reply.<br>
<br>
1. How can I update the eventlet version. I installed swift from
source (git). Pulling the latest code helps?<br>
2. Yes. Recently my clients changed to chunked encoding for
transfer. Are you saying chunked encoding is not supported by
swift?<br>
3. Yes, the 408s have made it to the clients from proxy servers.<br>
<br>
Regards,<br>
Shyam<br>
<br>
On Friday 09 May 2014 10:45 PM, Clay Gerrard wrote:<br>
</div>
<blockquote
cite="mid:CA+_JKzq6so3SVUjwRi+1nA_34aPOHtpWQMYnsL7guQCswNrf8Q@mail.gmail.com"
type="cite">
<div dir="ltr">I thought those tracebacks only showed up with old
versions of eventlet or and eventlet_debug = true?
<div><br>
</div>
<div>In my experience that normally indicates a client
disconnect on a chucked encoding transfer request (request w/o
a content-length). Do you know if your clients are using
transfer encoding chunked?</div>
<div><br>
</div>
<div>Are you seeing the 408 make it's way out to the client? It
wasn't clear to me if you only see these tracebacks on the
object-servers or in the proxy logs as well? Perhaps only one
of the three disks involved in the PUT are timing out and the
client still gets a successful response?</div>
<div><br>
</div>
<div>As the disks fill up replication and auditing is going to
consume more disk resources - you may have to tune the
concurrency and rate settings on those daemons. If the errors
happen consistently you could try running with background
consistency processes temporarily disabled and rule out if
they're causing disk contention on your setup with your
config.</div>
<div><br>
</div>
<div>-Clay</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 9, 2014 at 8:54 AM, Ben
Nemec <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">This is a
development list, and your question sounds more
usage-related. Please ask your question on the users list:
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
Thanks.<br>
<br>
-Ben
<div>
<div class="h5"><br>
<br>
On 05/09/2014 06:57 AM, Shyam Prasad N wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="h5">
Hi,<br>
<br>
I have a two node swift cluster receiving continuous
traffic (mostly<br>
overwrites for existing objects) of 1GB files each.<br>
<br>
Soon after the traffic started, I'm seeing the
following traceback from<br>
some transactions...<br>
Traceback (most recent call last):<br>
File "/home/eightkpc/swift/swift/proxy/controllers/obj.py",
line 692,<br>
in PUT<br>
chunk = next(data_source)<br>
File "/home/eightkpc/swift/swift/proxy/controllers/obj.py",
line 559,<br>
in <lambda><br>
data_source = iter(lambda:
reader(self.app.client_chunk_size), '')<br>
File "/home/eightkpc/swift/swift/common/utils.py",
line 2362, in read<br>
chunk = self.wsgi_input.read(*args, **kwargs)<br>
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py",
line 147,<br>
in read<br>
return self._chunked_read(self.rfile, length)<br>
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py",
line 137,<br>
in _chunked_read<br>
self.chunk_length =
int(rfile.readline().split(";", 1)[0], 16)<br>
ValueError: invalid literal for int() with base 16: ''
(txn:<br>
tx14e2df7680fd472fb92f0-00536ca4f0) (client_ip:
10.3.0.101)<br>
<br>
Seeing the following errors on storage logs...<br>
object-server: 10.3.0.102 - - [09/May/2014:01:36:49
+0000] "PUT<br>
/xvdg/492/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data"<br>
408 - "PUT<br>
<a moz-do-not-send="true"
href="http://10.3.0.102:8080/v1/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data"
target="_blank">http://10.3.0.102:8080/v1/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data</a>"<br>
"txf3b4e5f677004474bbd2f-00536c30d1" "proxy-server
12241" 95.6405 "-"<br>
<br>
It's success sometimes, but mostly 408 errors. I don't
see any other<br>
logs for the transaction ID. or around these 408
errors in the log<br>
files. Is this a disk timeout issue? These are only
1GB files and normal<br>
writes to files on these disks are quite fast.<br>
<br>
The timeouts from the swift proxy files are...<br>
root@bulkstore-112:~# grep -R timeout /etc/swift/*<br>
/etc/swift/proxy-server.conf:client_timeout = 600<br>
/etc/swift/proxy-server.conf:node_timeout = 600<br>
/etc/swift/proxy-server.conf:recoverable_node_timeout
= 600<br>
<br>
Can someone help me troubleshoot this issue?<br>
<br>
--<br>
-Shyam<br>
<br>
<br>
</div>
</div>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>