Open Stack

Hi,

I have a two node swift cluster receiving continuous traffic (mostly
overwrites for existing objects) of 1GB files each.

Soon after the traffic started, I'm seeing the following traceback from
some transactions...
Traceback (most recent call last):
  File "/home/eightkpc/swift/swift/proxy/controllers/obj.py", line 692, in
PUT
    chunk = next(data_source)
  File "/home/eightkpc/swift/swift/proxy/controllers/obj.py", line 559, in
<lambda>
    data_source = iter(lambda: reader(self.app.client_chunk_size), '')
  File "/home/eightkpc/swift/swift/common/utils.py", line 2362, in read
    chunk = self.wsgi_input.read(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 147, in
read
    return self._chunked_read(self.rfile, length)
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 137, in
_chunked_read
    self.chunk_length = int(rfile.readline().split(";", 1)[0], 16)
ValueError: invalid literal for int() with base 16: '' (txn:
tx14e2df7680fd472fb92f0-00536ca4f0) (client_ip: 10.3.0.101)

Seeing the following errors on storage logs...
object-server: 10.3.0.102 - - [09/May/2014:01:36:49 +0000] "PUT
/xvdg/492/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data"
408 - "PUT
http://10.3.0.102:8080/v1/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data"
"txf3b4e5f677004474bbd2f-00536c30d1" "proxy-server 12241" 95.6405 "-"

It's success sometimes, but mostly 408 errors. I don't see any other logs
for the transaction ID. or around these 408 errors in the log files. Is
this a disk timeout issue? These are only 1GB files and normal writes to
files on these disks are quite fast.

The timeouts from the swift proxy files are...
root at bulkstore-112:~# grep -R timeout /etc/swift/*
/etc/swift/proxy-server.conf:client_timeout = 600
/etc/swift/proxy-server.conf:node_timeout = 600
/etc/swift/proxy-server.conf:recoverable_node_timeout = 600

Can someone help me troubleshoot this issue?

-- 
-Shyam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140509/64dbd7b8/attachment.html>