Open Stack

Vivek,

The problem is swift client is only downloading a chunk of object not
the whole object so the etag mismatch. Could you paste the value of
'rgw_max_chunk_size'. Please be sure you set this to a sane
value(<4MB, atleast for Giant release this works below this value).



On Tue, Dec 16, 2014 at 12:26 PM, Vivek Varghese Cherian
<vivekcherian at gmail.com> wrote:
> Hi,
>
> I am integrating ceph firefly radosgw with openstack juno keystone, the
> operating system used on the ceph
> nodes and on the openstack node is Ubuntu 14.04.
>
> I am able to create containers and upload files using the swift client to
> ceph.
>
> But when I try to download files, I am getting the following error,
>
> ppmuser at ppm-dc-c3sv3-ju:~$ swift --verbose --debug -V 1 -A
> http://10.x.x.126/auth -U swift:swift -K <Key_here> download demo-container1
> file_12345.txt
>
> INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
> DEBUG:urllib3.connectionpool:Setting read timeout to None
> DEBUG:urllib3.connectionpool:"GET /auth HTTP/1.1" 204 0
> DEBUG:swiftclient:REQ: curl -i http://10.x.x.126/auth -X GET
> DEBUG:swiftclient:RESP STATUS: 204 No Content
> DEBUG:swiftclient:RESP HEADERS: [('x-auth-token',
> 'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'),
> ('x-storage-token',
> 'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'),
> ('date', 'Tue, 16 Dec 2014 05:49:11 GMT'), ('x-storage-url',
> 'http://10.x.x.126/swift/v1'), ('server', 'Apache/2.4.7 (Ubuntu)'),
> ('content-type', 'application/json')]
> INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
> DEBUG:urllib3.connectionpool:Setting read timeout to None
> DEBUG:urllib3.connectionpool:"GET /swift/v1/demo-container1/file_12345.txt
> HTTP/1.1" 200 14
> DEBUG:swiftclient:REQ: curl -i
> http://10.x.x.126/swift/v1/demo-container1/file_12345.txt -X GET -H
> "X-Auth-Token:
> AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e"
> DEBUG:swiftclient:RESP STATUS: 200 OK
> DEBUG:swiftclient:RESP HEADERS: [('content-length', '14'), ('accept-ranges',
> 'bytes'), ('server', 'Apache/2.4.7 (Ubuntu)'), ('last-modified', 'Mon, 15
> Dec 2014 13:35:37 GMT'), ('etag', '94b371af646d4c53d8f3f8ff7be74dbb'),
> ('date', 'Tue, 16 Dec 2014 05:49:11 GMT'), ('x-object-meta-mtime',
> '1418650375.550963')]
>
> Error downloading object 'demo-container1/file_12345.txt': 'Error
> downloading file_12345.txt: read_length != content_length, 0 != 14'
>
> ppmuser at ppm-dc-c3sv3-ju:~
>
>
> The apache2 error and access logs on the radosgw ceph node is as follows,
>
> root at ppm-c240-ceph3:~# tail -f /var/log/apache2/error.log
>
> [Tue Dec 16 01:10:20.735061 2014] [fastcgi:error] [pid 31426:tid
> 140231210624768] [client 10.x.x.175:55902] FastCGI: comm with server
> "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
> [Tue Dec 16 01:10:20.735134 2014] [fastcgi:error] [pid 31426:tid
> 140231210624768] [client 10.x.x.175:55902] FastCGI: incomplete headers (0
> bytes) received from server "/var/www/s3gw.fcgi"
> [Tue Dec 16 01:10:20.735161 2014] [http:error] [pid 31426:tid
> 140231210624768] [client 10.x.x.175:55902] not unsetting Content-Length on
> HEAD response (rgw changes)\n
> [Tue Dec 16 01:10:20.857215 2014] [:warn] [pid 31426:tid 140231202232064]
> FastCGI: 10.x.x.175 HEAD
> http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93
> auth
> [Tue Dec 16 01:10:50.887381 2014] [fastcgi:error] [pid 31426:tid
> 140231202232064] [client 10.x.x.175:55910] FastCGI: comm with server
> "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
> [Tue Dec 16 01:10:50.887454 2014] [fastcgi:error] [pid 31426:tid
> 140231202232064] [client 10.x.x.175:55910] FastCGI: incomplete headers (0
> bytes) received from server "/var/www/s3gw.fcgi"
> [Tue Dec 16 01:10:50.887481 2014] [http:error] [pid 31426:tid
> 140231202232064] [client 10.x.x.175:55910] not unsetting Content-Length on
> HEAD response (rgw changes)\n
> [Tue Dec 16 01:10:50.898281 2014] [:warn] [pid 31426:tid 140231193839360]
> FastCGI: 10.x.x.175 GET
> http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93
> auth
> [Tue Dec 16 01:11:20.928476 2014] [fastcgi:error] [pid 31426:tid
> 140231193839360] [client 10.x.x.175:55911] FastCGI: comm with server
> "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
> [Tue Dec 16 01:11:20.928556 2014] [fastcgi:error] [pid 31426:tid
> 140231193839360] [client 10.x.x.175:55911] FastCGI: incomplete headers (0
> bytes) received from server "/var/www/s3gw.fcgi"
>
> root at ppm-c240-ceph3:~# tail -f /var/log/apache2/access.log
>
> 10.81.83.175 - - [16/Dec/2014:01:00:20 -0500] "HEAD
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
> "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:00:50 -0500] "GET
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
> 719 "-" "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET /auth HTTP/1.1" 204 431
> "-" "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET
> /swift/v1/demo-container1?format=json HTTP/1.1" 200 411 "-"
> "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET
> /swift/v1/demo-container1?format=json&marker=file_12345.txt HTTP/1.1" 200
> 174 "-" "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:08:50 -0500] "HEAD
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
> "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:09:20 -0500] "GET
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
> 719 "-" "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:09:50 -0500] "HEAD
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
> "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:10:20 -0500] "HEAD
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
> "python-swiftclient-2.3.0"
> 10.81.83.175 - - [16/Dec/2014:01:10:50 -0500] "GET
> /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
> 719 "-" "python-swiftclient-2.3.0"
>
>
> The radosgw logs is as follows,
>
> root at ppm-c240-ceph3:/var/log/radosgw# tail -f ceph-client.admin
>
> 2014-12-09 11:39:19.862854 7fdbfa1257c0  0 librados: client.admin
> initialization error (2) No such file or directory
> 2014-12-09 11:39:19.863429 7fdbfa1257c0 -1 Couldn't init storage provider
> (RADOS)
> 2014-12-09 11:41:25.894934 7ffa8fe087c0  0 ceph version 0.80.7
> (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5843
> 2014-12-09 11:41:25.922673 7ffa8fe087c0 -1 monclient(hunting): ERROR:
> missing keyring, cannot use cephx for authentication
> 2014-12-09 11:41:25.922807 7ffa8fe087c0  0 librados: client.admin
> initialization error (2) No such file or directory
> 2014-12-09 11:41:25.924901 7ffa8fe087c0 -1 Couldn't init storage provider
> (RADOS)
> 2014-12-09 11:42:12.110334 7f145f77d7c0  0 ceph version 0.80.7
> (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5862
> 2014-12-09 11:42:12.139115 7f145f77d7c0 -1 monclient(hunting): ERROR:
> missing keyring, cannot use cephx for authentication
> 2014-12-09 11:42:12.139215 7f145f77d7c0  0 librados: client.admin
> initialization error (2) No such file or directory
> 2014-12-09 11:42:12.141559 7f145f77d7c0 -1 Couldn't init storage provider
> (RADOS)
>
>
> Any pointers as to what could be the cause of the error in downloading files
> from the ceph cluster using the openstack juno swift client would be highly
> appreciated.
>
>
> Regards,
> ---
> Vivek Varghese Cherian
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
-Pushpesh