[openstack-dev] [swift] 503 Service Unavailable errors

Snider, Tim Tim.Snider at netapp.com
Mon Sep 16 12:35:47 UTC 2013 

When I'm doing large transfers Swift often returns 503 errors with "proxy-server Object PUT exceptions during send, 1/2 required connections" in the log file.
Is this an indication of network issues or can someone explain the cause and possible solution?
Thanks

*   Trying 192.168.10.90...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
> PUT /v1/AUTH_test/load/1gbfile51_0 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 192.168.10.90:8080
> Accept: */*
> X-Auth-Token: AUTH_tk422c579f97bb4da69528820add184204
> Content-Length: 1073741824
>
} [data not shown]
  0 1024M    0     0    0 3040k      0  3304k  0:05:17 --:--:--  0:05:17 3307k
  0 1024M    0     0    0 4512k      0  2058k  0:08:29  0:00:02  0:08:27 2060k
  0 1024M    0     0    0 5200k      0  1772k  0:09:51  0:00:02  0:09:49 1772k
  0 1024M    0     0    0 6544k      0  1668k  0:10:28  0:00:03  0:10:25 1669k
  0 1024M    0     0    0 8960k      0  1835k  0:09:31  0:00:04  0:09:27 1835k
  1 1024M    0     0    1 12.7M      0  2215k  0:07:53  0:00:05  0:07:48 2012k
  1 1024M    0     0    1 19.2M      0  2859k  0:06:06  0:00:06  0:06:00 3233k
  2 1024M    0     0    2 22.0M      0  2768k  0:06:18  0:00:08  0:06:10 3327k
  2 1024M    0     0    2 26.4M      0  3058k  0:05:42  0:00:08  0:05:34 4159k
  2 1024M    0     0    2 29.5M      0  2905k  0:06:00  0:00:10  0:05:50 3851k
  2 1024M    0     0    2 29.5M      0  2650k  0:06:35  0:00:11  0:06:24 3113k
  2 1024M    0     0    2 29.5M      0  2436k  0:07:10  0:00:12  0:06:58 1907k
  2 1024M    0     0    2 29.5M      0  2254k  0:07:45  0:00:13  0:07:32 1455k
  2 1024M    0     0    2 29.5M      0  2097k  0:08:19  0:00:14  0:08:05  560k
  2 1024M    0     0    2 29.5M      0  1961k  0:08:54  0:00:15  0:08:39     0
  2 1024M    0     0    2 29.5M      0  1841k  0:09:29  0:00:16  0:09:13     0
  2 1024M    0     0    2 29.5M      0  1736k  0:10:04  0:00:17  0:09:47     0
  2 1024M    0     0    2 29.5M      0  1641k  0:10:38  0:00:18  0:10:20     0< HTTP/1.1 503 Service Unavailable
< Content-Length: 212
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx3c2f6133fc2e4b43bebc939aea2ae17f
< Date: Mon, 16 Sep 2013 02:02:37 GMT
* HTTP error before end of send, stop sending
<
{ [data not shown]
  2 1024M  100   212    2 29.5M     11  1586k  0:11:00  0:00:19  0:10:41     0
  2 1024M  100   212    2 29.5M     11  1586k  0:11:00  0:00:19  0:10:41     0
* Closing connection #0
<html>
<head>
  <title>503 Service Unavailable</title>
</head>
<body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />
</body>


ssh -i /root/.ssh/id_rsa  root at 10.113.193.90 grep tx3c2f6133fc2e4b43bebc939aea2ae17f /var/log/swift/*
/var/log/swift/proxy.error:Sep 15 19:02:37 swift14 proxy-server Object PUT exceptions during send, 1/2 required connections (txn: tx3c2f6133fc2e4b43bebc939aea2ae17f) (client_ip: 192.168.10.69)
/var/log/swift/proxy.log:Sep 15 19:02:37 swift14 proxy-server 192.168.10.69 192.168.10.69 16/Sep/2013/02/02/37 PUT /v1/AUTH_test/load/1gbfile51_0 HTTP/1.0 503 - curl/7.22.0%20%28x86_64-pc-linux-gnu%29%20libcurl/7.22.0%20OpenSSL/1.0.1%20zlib/1.2.3.4%20libidn/1.23%20librtmp/2.3 test%2CAUTH_tk422c579f97bb4da69528820add184204 29556736 212 - tx3c2f6133fc2e4b43bebc939aea2ae17f - 19.0232 -


Thanks,
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130916/0e1bb25a/attachment.html>

More information about the OpenStack-dev mailing list