Open Stack

Dear all,

I run the stable branch of OpenStack Icehouse on Scientific Linux 6,
installed with Packstack, with Trove release 2014.2.b2 and
python-troveclient 1.0.5 (newest release/tag for both of them).

My problem: if as OpenStack user `admin`, that means, with sourced
`keystonerc_admin`, I run `trove list` or `trove datastore-list` I get

> ERROR: Unauthorized (HTTP 401).

The same happens with user `trove` (see below).

The log files trove.log and keystone.log show (full excerpts below):

trove.log:
> * Unexpected response from keystone service: {u'error': {u'message': u"object of type 'NoneType' has no len()", u'code': 400, u'title': u'Bad Request'}}
> * ServiceError: invalid json response
> * Authorization failed for token

keystone.log:
> * TypeError: object of type 'NoneType' has no len()

I think it is a problem with the users and tenants I configured for
Trove. Someone having the same problem with Glance could fix it by
putting the right login information into glance.conf, see
https://answers.launchpad.net/glance/+question/229769 .

For configuration of Trove I followed the OpenStack documentation [1],
Troves documentation for manual install [2] and the DevStack code [3].

[1]
http://docs.openstack.org/icehouse/install-guide/install/yum/content/trove-install.html
[2] http://docs.openstack.org/developer/trove/dev/manual_install.html
[3] https://github.com/openstack-dev/devstack/blob/stable/icehouse/lib/trove

The three sources use different combinations of users and tenants. Does
someone of you know which users and tenants have to be used? At the
moment, I have the following configuration:

Users and tenants:
* tenant `trove`
* user `trove` is member and admin in tenant `trove` and `services`,
which is the service tenant in my installation of OpenStack
* user `admin` is member and admin in tenant `trove`, and for testing
even member and admin in `services`, but this didn't help

Trove's api-paste.ini:
> [filter:authtoken]
> admin_user=trove
> admin_password=***
> admin_tenant_name=services

trove-taskmanager.conf, trove-conductor.conf and trove-guestagent.conf:
> [DEFAULT]
> nova_proxy_admin_user=admin
> nova_proxy_admin_pass=***
> nova_proxy_admin_tenant_name=trove

[1] uses tenant `services` here.

Any help and guidelines are appreciated.


Best regards,
Benjamin

### trove.log ###

> 2014-08-21 14:55:27.122 6221 INFO eventlet.wsgi [-] (6221) accepted ('***.***.***.***', 45415)
> 2014-08-21 14:55:27.123 6221 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:666
> 2014-08-21 14:55:27.124 6221 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:725
> 2014-08-21 14:55:27.132 6221 WARNING keystoneclient.middleware.auth_token [-] Unexpected response from keystone service: {u'error': {u'message': u"object of type 'NoneType' has no len()", u'code': 400, u'title': u'Bad Request'}}
> 2014-08-21 14:55:27.132 6221 DEBUG keystoneclient.middleware.auth_token [-] Token validation failure. _validate_user_token /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:943
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token Traceback (most recent call last):
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 930, in _validate_user_token
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     verified = self.verify_signed_token(user_token, token_ids)
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1347, in verify_signed_token
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     if self.is_signed_token_revoked(token_ids):
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1299, in is_signed_token_revoked
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     if self._is_token_id_in_revoked_list(token_id):
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1306, in _is_token_id_in_revoked_list
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     revocation_list = self.token_revocation_list
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1413, in token_revocation_list
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     self.token_revocation_list = self.fetch_revocation_list()
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1446, in fetch_revocation_list
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     headers = {'X-Auth-Token': self.get_admin_token()}
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 777, in get_admin_token
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     self.admin_token_expiry) = self._request_admin_token()
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 890, in _request_admin_token
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token     raise ServiceError('invalid json response')
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token ServiceError: invalid json response
> 2014-08-21 14:55:27.132 6221 TRACE keystoneclient.middleware.auth_token
> 2014-08-21 14:55:27.134 6221 DEBUG keystoneclient.middleware.auth_token [-] Marking token as unauthorized in cache _cache_store_invalid /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:1239
> 2014-08-21 14:55:27.134 6221 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
> 2014-08-21 14:55:27.135 6221 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
> 2014-08-21 14:55:27.135 6221 INFO eventlet.wsgi [-] ***.***.***.*** - - [21/Aug/2014 14:55:27] "GET /v1.0/b426bf07c5cd48d1b2525699bca29cdb/instances HTTP/1.1" 401 199 0.011922

### keystone.log ###

> 2014-08-21 14:55:27.100 3165 INFO eventlet.wsgi.server [-] ***.***.***.*** - - [21/Aug/2014 14:55:27] "POST /v2.0/tokens HTTP/1.1" 200 10323 0.213025
> 2014-08-21 14:55:27.129 3165 ERROR keystone.common.wsgi [-] object of type 'NoneType' has no len()
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi Traceback (most recent call last):
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 207, in __call__
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi     result = method(context, **params)
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 98, in authenticate
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi     context, auth)
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 256, in _authenticate_local
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi     if len(username) > CONF.max_param_size:
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi TypeError: object of type 'NoneType' has no len()
> 2014-08-21 14:55:27.129 3165 TRACE keystone.common.wsgi
> 2014-08-21 14:55:27.131 3165 INFO eventlet.wsgi.server [-] ***.***.***.*** - - [21/Aug/2014 14:55:27] "POST /v2.0/tokens HTTP/1.1" 400 239 0.004268