Open Stack

Hi Y'all,

I'm trying to configure Glance on RedHat RDO Icehouse, but I'm getting 
an auth error when I try to upload an image to it.  On the client I'm 
trying to upload from, I see:

# glance -d image-create --name="CirrOS 0.3.1" --disk-format=qcow2 
--container-format=bare --is-public=true < cirros-0.3.1-x86_64-disk.img
curl -i -X POST -H 'x-image-meta-container_format: bare' -H 
'Transfer-Encoding: chunked' -H 'User-Agent: python-glanceclient' -H 
'x-image-meta-size: 13147648' -H 'x-image-meta-is_public: True' -H 
'X-Auth-Token: <...removed token...>' -H 'Content-Type: 
application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 
'x-image-meta-name: CirrOS 0.3.1' -d '<open file '<stdin>', mode 'r' at 
0x7f49edd5d0c0>' https://my-public-server.com:9292/v1/images

HTTP/1.1 500 Internal Server Error
date: Mon, 24 Mar 2014 18:34:03 GMT
content-length: 0
content-type: text/plain
connection: close

Request returned failure status.
HTTPInternalServerError (HTTP 500)

I've launched glance-api in debug mode on the server side, and I see 
this when the above command is run:

2014-03-24 11:36:14.202 14543 DEBUG 
glance.api.middleware.version_negotiation [-] Determining version of 
request: POST /v1/images Accept:  process_request 
/usr/lib/python2.6/site-packages/glance/api/middleware/version_negotiation.py:44
2014-03-24 11:36:14.203 14543 DEBUG 
glance.api.middleware.version_negotiation [-] Using url versioning 
process_request 
/usr/lib/python2.6/site-packages/glance/api/middleware/version_negotiation.py:57
2014-03-24 11:36:14.203 14543 DEBUG 
glance.api.middleware.version_negotiation [-] Matched version: v1 
process_request 
/usr/lib/python2.6/site-packages/glance/api/middleware/version_negotiation.py:69
2014-03-24 11:36:14.204 14543 DEBUG 
glance.api.middleware.version_negotiation [-] new path /v1/images 
process_request 
/usr/lib/python2.6/site-packages/glance/api/middleware/version_negotiation.py:70
2014-03-24 11:36:14.204 14543 DEBUG keystoneclient.middleware.auth_token 
[-] Authenticating user token __call__ 
/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:558
2014-03-24 11:36:14.205 14543 DEBUG keystoneclient.middleware.auth_token 
[-] Removing headers from request environment: 
X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role 
_remove_auth_headers 
/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:617
2014-03-24 11:36:14.226 14543 INFO urllib3.connectionpool [-] Starting 
new HTTP connection (1): genome-cloud-0-10.kilokluster.ucsc.edu
2014-03-24 11:36:14.339 14543 DEBUG urllib3.connectionpool [-] "POST 
/v2.0/tokens HTTP/1.1" 200 3446 _make_request 
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-03-24 11:36:14.382 14543 INFO urllib3.connectionpool [-] Starting 
new HTTP connection (1): genome-cloud-0-10.kilokluster.ucsc.edu
2014-03-24 11:36:14.422 14543 DEBUG urllib3.connectionpool [-] "GET 
/v2.0/tokens/revoked HTTP/1.1" 200 686 _make_request 
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-03-24 11:36:14.433 14543 INFO urllib3.connectionpool [-] Starting 
new HTTP connection (1): genome-cloud-0-10.kilokluster.ucsc.edu
2014-03-24 11:36:14.439 14543 DEBUG urllib3.connectionpool [-] "GET 
/v2.0/certificates/signing HTTP/1.1" 200 4251 _make_request 
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-03-24 11:36:14.451 14543 INFO urllib3.connectionpool [-] Starting 
new HTTP connection (1): genome-cloud-0-10.kilokluster.ucsc.edu
2014-03-24 11:36:14.455 14543 DEBUG urllib3.connectionpool [-] "GET 
/v2.0/certificates/ca HTTP/1.1" 200 1277 _make_request 
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2014-03-24 11:36:14.476 14543 DEBUG keystoneclient.middleware.auth_token 
[-] Storing 326d8c391f19d07c9f5a69d40da33f0a token in memcache 
_cache_put 
/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:1061
2014-03-24 11:36:14.477 14543 DEBUG keystoneclient.middleware.auth_token 
[-] Received request from user: f8fdf7f84ad34c439c4075b5e3720211 with 
project_id : f7e61747885045d8b266a161310c0094 and roles: _member_ 
_build_user_headers 
/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:922
2014-03-24 11:36:14.487 14543 DEBUG routes.middleware [-] Matched POST 
/images __call__ 
/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py:100
2014-03-24 11:36:14.487 14543 DEBUG routes.middleware [-] Route path: 
'/images', defaults: {'action': u'create', 'controller': 
<glance.common.wsgi.Resource object at 0x34c7450>} __call__ 
/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py:102
2014-03-24 11:36:14.487 14543 DEBUG routes.middleware [-] Match dict: 
{'action': u'create', 'controller': <glance.common.wsgi.Resource object 
at 0x34c7450>} __call__ 
/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py:103
2014-03-24 11:36:14.488 14543 DEBUG glance.registry.client.v1.api 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] Adding image metadata... 
add_image_metadata 
/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py:159
2014-03-24 11:36:14.488 14543 DEBUG glance.common.client 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] Constructed URL: 
http://0.0.0.0:9191/images _construct_url 
/usr/lib/python2.6/site-packages/glance/common/client.py:407
2014-03-24 11:36:14.556 14543 DEBUG glance.common.client 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] Constructed URL: 
http://0.0.0.0:9191/images _construct_url 
/usr/lib/python2.6/site-packages/glance/common/client.py:407
2014-03-24 11:36:14.560 14543 INFO glance.registry.client.v1.client 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] Registry client request POST /images 
raised NotAuthenticated
2014-03-24 11:36:14.564 14543 INFO glance.wsgi.server 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 382, 
in handle_one_response
     result = self.application(self.environ, start_response)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in 
__call__
     resp = self.call_func(req, *args, **self.kwargs)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in 
call_func
     return self.func(req, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/wsgi.py", line 
372, in __call__
     response = req.get_response(self.application)
   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, 
in send
     application, catch_exc_info=False)
   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, 
in call_application
     app_iter = application(self.environ, start_response)
   File 
"/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", 
line 571, in __call__
     return self.app(env, start_response)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in 
__call__
     resp = self.call_func(req, *args, **self.kwargs)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in 
call_func
     return self.func(req, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/wsgi.py", line 
372, in __call__
     response = req.get_response(self.application)
   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, 
in send
     application, catch_exc_info=False)
   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, 
in call_application
     app_iter = application(self.environ, start_response)
   File "/usr/lib/python2.6/site-packages/paste/urlmap.py", line 203, in 
__call__
     return app(environ, start_response)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in 
__call__
     return resp(environ, start_response)
   File 
"/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", 
line 131, in __call__
     response = self.app(environ, start_response)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in 
__call__
     return resp(environ, start_response)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in 
__call__
     resp = self.call_func(req, *args, **self.kwargs)
   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in 
call_func
     return self.func(req, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/wsgi.py", line 
604, in __call__
     request, **action_args)
   File "/usr/lib/python2.6/site-packages/glance/common/wsgi.py", line 
623, in dispatch
     return method(*args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/utils.py", line 
435, in wrapped
     return func(self, req, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/api/v1/images.py", line 
781, in create
     image_meta = self._reserve(req, image_meta)
   File "/usr/lib/python2.6/site-packages/glance/api/v1/images.py", line 
514, in _reserve
     image_meta = registry.add_image_metadata(req.context, image_meta)
   File 
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", 
line 161, in add_image_metadata
     return c.add_image(image_meta)
   File 
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", 
line 163, in add_image
     res = self.do_request("POST", "/images", body=body, headers=headers)
   File 
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", 
line 107, in do_request
     **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/client.py", line 
65, in wrapped
     return func(self, *args, **kwargs)
   File "/usr/lib/python2.6/site-packages/glance/common/client.py", line 
382, in do_request
     headers=copy.deepcopy(headers))
   File "/usr/lib/python2.6/site-packages/glance/common/client.py", line 
79, in wrapped
     return func(self, method, url, body, headers)
   File "/usr/lib/python2.6/site-packages/glance/common/client.py", line 
523, in _do_request
     raise exception.NotAuthenticated(res.read())
NotAuthenticated: Authentication required


2014-03-24 11:36:14.967 14543 INFO glance.wsgi.server 
[3f58e73a-6eb0-4747-ab61-e8b81fbe55d3 f8fdf7f84ad34c439c4075b5e3720211 
f7e61747885045d8b266a161310c0094] 111.213.225.79,10.1.1.137 - - 
[24/Mar/2014 11:36:14] "POST /v1/images HTTP/1.1" 500 139 0.765716

So I see some Auth errors in that, but I can't tell _what_ kind of Auth 
errors they are.  User auth errors from the user uploading the file? 
Service Auth errors from the glance service trying to auth to keystone? 
  QPID auth errors?

Can anyone see what's wrong?  Then I can better debug where my problem 
is...  I've confirmed the user can auth ok with "keystone token-get'", 
that seems OK, I have the service user in keystone, not sure where it's 
failing...

keystone logs don't really show anything other than:

2014-03-24 11:41:52.420 16503 WARNING keystone.common.wsgi [-] 
Authorization failed. The request you have made requires authentication. 
from 10.1.1.148

Where 10.1.1.148 is the glance-api server on my internal network.

Thanks for any hints!!

-erich