Open Stack

Hi,

I am integrating ceph firefly radosgw with openstack juno keystone, the
operating system used on the ceph
nodes and on the openstack node is Ubuntu 14.04.

I am able to create containers and upload files using the swift client to
ceph.

But when I try to download files, I am getting the following error,

ppmuser at ppm-dc-c3sv3-ju:~$ swift --verbose --debug -V 1 -A
http://10.x.x.126/auth -U swift:swift -K <Key_here> download
demo-container1 file_12345.txt

INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /auth HTTP/1.1" 204 0
DEBUG:swiftclient:REQ: curl -i http://10.x.x.126/auth -X GET
DEBUG:swiftclient:RESP STATUS: 204 No Content
DEBUG:swiftclient:RESP HEADERS: [('x-auth-token',
'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'),
('x-storage-token',
'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'),
('date', 'Tue, 16 Dec 2014 05:49:11 GMT'), ('x-storage-url', '
http://10.x.x.126/swift/v1'), ('server', 'Apache/2.4.7 (Ubuntu)'),
('content-type', 'application/json')]
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /swift/v1/demo-container1/file_12345.txt
HTTP/1.1" 200 14
DEBUG:swiftclient:REQ: curl -i
http://10.x.x.126/swift/v1/demo-container1/file_12345.txt -X GET -H
"X-Auth-Token:
AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: [('content-length', '14'),
('accept-ranges', 'bytes'), ('server', 'Apache/2.4.7 (Ubuntu)'),
('last-modified', 'Mon, 15 Dec 2014 13:35:37 GMT'), ('etag',
'94b371af646d4c53d8f3f8ff7be74dbb'), ('date', 'Tue, 16 Dec 2014 05:49:11
GMT'), ('x-object-meta-mtime', '1418650375.550963')]

Error downloading object 'demo-container1/file_12345.txt': 'Error
downloading file_12345.txt: read_length != content_length, 0 != 14'

ppmuser at ppm-dc-c3sv3-ju:~


The apache2 error and access logs on the radosgw ceph node is as follows,

root at ppm-c240-ceph3:~# tail -f /var/log/apache2/error.log

[Tue Dec 16 01:10:20.735061 2014] [fastcgi:error] [pid 31426:tid
140231210624768] [client 10.x.x.175:55902] FastCGI: comm with server
"/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:10:20.735134 2014] [fastcgi:error] [pid 31426:tid
140231210624768] [client 10.x.x.175:55902] FastCGI: incomplete headers (0
bytes) received from server "/var/www/s3gw.fcgi"
[Tue Dec 16 01:10:20.735161 2014] [http:error] [pid 31426:tid
140231210624768] [client 10.x.x.175:55902] not unsetting Content-Length on
HEAD response (rgw changes)\n
[Tue Dec 16 01:10:20.857215 2014] [:warn] [pid 31426:tid 140231202232064]
FastCGI: 10.x.x.175 HEAD
http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93
auth
[Tue Dec 16 01:10:50.887381 2014] [fastcgi:error] [pid 31426:tid
140231202232064] [client 10.x.x.175:55910] FastCGI: comm with server
"/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:10:50.887454 2014] [fastcgi:error] [pid 31426:tid
140231202232064] [client 10.x.x.175:55910] FastCGI: incomplete headers (0
bytes) received from server "/var/www/s3gw.fcgi"
[Tue Dec 16 01:10:50.887481 2014] [http:error] [pid 31426:tid
140231202232064] [client 10.x.x.175:55910] not unsetting Content-Length on
HEAD response (rgw changes)\n
[Tue Dec 16 01:10:50.898281 2014] [:warn] [pid 31426:tid 140231193839360]
FastCGI: 10.x.x.175 GET
http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93
auth
[Tue Dec 16 01:11:20.928476 2014] [fastcgi:error] [pid 31426:tid
140231193839360] [client 10.x.x.175:55911] FastCGI: comm with server
"/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:11:20.928556 2014] [fastcgi:error] [pid 31426:tid
140231193839360] [client 10.x.x.175:55911] FastCGI: incomplete headers (0
bytes) received from server "/var/www/s3gw.fcgi"

root at ppm-c240-ceph3:~# tail -f /var/log/apache2/access.log

10.81.83.175 - - [16/Dec/2014:01:00:20 -0500] "HEAD
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
"python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:00:50 -0500] "GET
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
719 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET /auth HTTP/1.1" 204 431
"-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET
/swift/v1/demo-container1?format=json HTTP/1.1" 200 411 "-"
"python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET
/swift/v1/demo-container1?format=json&marker=file_12345.txt HTTP/1.1" 200
174 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:50 -0500] "HEAD
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
"python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:09:20 -0500] "GET
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
719 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:09:50 -0500] "HEAD
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
"python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:10:20 -0500] "HEAD
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-"
"python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:10:50 -0500] "GET
/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500
719 "-" "python-swiftclient-2.3.0"


The radosgw logs is as follows,

root at ppm-c240-ceph3:/var/log/radosgw# tail -f ceph-client.admin

2014-12-09 11:39:19.862854 7fdbfa1257c0  0 librados: client.admin
initialization error (2) No such file or directory
2014-12-09 11:39:19.863429 7fdbfa1257c0 -1 Couldn't init storage provider
(RADOS)
2014-12-09 11:41:25.894934 7ffa8fe087c0  0 ceph version 0.80.7
(6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5843
2014-12-09 11:41:25.922673 7ffa8fe087c0 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2014-12-09 11:41:25.922807 7ffa8fe087c0  0 librados: client.admin
initialization error (2) No such file or directory
2014-12-09 11:41:25.924901 7ffa8fe087c0 -1 Couldn't init storage provider
(RADOS)
2014-12-09 11:42:12.110334 7f145f77d7c0  0 ceph version 0.80.7
(6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5862
2014-12-09 11:42:12.139115 7f145f77d7c0 -1 monclient(hunting): ERROR:
missing keyring, cannot use cephx for authentication
2014-12-09 11:42:12.139215 7f145f77d7c0  0 librados: client.admin
initialization error (2) No such file or directory
2014-12-09 11:42:12.141559 7f145f77d7c0 -1 Couldn't init storage provider
(RADOS)


Any pointers as to what could be the cause of the error in downloading
files from the ceph cluster using the openstack juno swift client would be
highly appreciated.


Regards,
---
Vivek Varghese Cherian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141216/4d370e20/attachment.html>