Open Stack

Hi

We are facing few issues with keystone access in our setups.

We have a setup which has three VMs. VM1 and VM3 are on same compute host whereas VM2 is on a different compute host.

- VM1 (mysite-ceilometer-6): Running Mitaka Ceilometer service + Keystone (v2) service
- VM2 (mysite-ceilometer-7): Running Ceilometer client
- VM3 (mysite-ceilometer-8): Running Ceilometer client

I have the keystone endpoints setup on VM1 as shown below: 'keystone endpoint-list'
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
|              id   |   region  |              publicurl               |             internalurl              |                adminurl               |            service_id            |
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
| 7c56f9e61 | RegionOne | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:35357/v2.0 | 4d871 |
| adc3ac02f | RegionOne |   http://mysite-ceilometer-6:8777    |   http://mysite-ceilometer-6:8777    |    http://mysite-ceilometer-6:8777    | 50fb869 |
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+

I have set the following OS variables in both VM2 and VM3:
OS_USERNAME="admin"
OS_PASSWORD="password"
OS_TENANT_NAME="admin"
OS_AUTH_URL="http://mysite-ceilometer-6:5000/v2.0"

The interesting observation I have made is that I am able to run the ceilometer or keystone commands successfully only from VM3 (which is on the same compute host as VM1) but not from VM2. Attached some of the logs I have collected during this observation. Can any of you help us with identifying what is going wrong here in this setup? Or any pointers on debugging why keystone/tokens REST API is not responding in some scenarios...? Thanks for ur help.

The output from VM3 (mysite-ceilometer-8):
---------------------------------
root at 69861113613c:/usr/local/share# ceilometer -d meter-list
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: req-da4329a5-0374-48de-9a07-229636042f64 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: req-491da351-24c1-4b4d-9558-b6d66a280c35 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
DEBUG (client) REQ: curl -g -i -X 'GET' 'http://mysite-ceilometer-6:8777/v2/meters' -H 'User-Agent: ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}a9....'
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2/meters HTTP/1.1" 200 2
DEBUG (client) RESP: [200] {'Date': 'Wed, 23 Nov 2016 22:49:29 GMT', 'Content-Length': '2', 'Content-Type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-6bd80a0c-db08-4907-8b6b-4d477a6d7514', 'Server': 'Werkzeug/0.9.4 Python/2.7.6'}

DEBUG (client) RESP BODY: []

+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

The output from VM2 (mysite-ceilometer-7):
---------------------------------

root at cadf877c4cdb:/usr/local/share# ceilometer -d meter-list
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:50:12 GMT x-openstack-request-id: req-f11122bc-bc3c-4521-81f3-d140bfce25b3 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
<<<<<Hangs here until time out>>>
Traceback (most recent call last):
  File "/usr/local/bin/ceilometer", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 292, in main
    CeilometerShell().main(args)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 238, in main
    client = ceiloclient.get_client(api_version, **client_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
    return Client(version, endpoint, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
    return client_class(*args, **client_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 67, in __init__
    self.alarm_client = self._get_alarm_client(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 122, in _get_alarm_client
    c.get("/")
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 244, in client_request
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 315, in authenticate
    self.auth_plugin.authenticate(self)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/auth.py", line 203, in authenticate
    self._do_authenticate(http_client)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 192, in _do_authenticate
    endpoint = _get_endpoint(ks_session, **ks_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 162, in _get_endpoint
    region_name=kwargs.get('region_name'))
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 664, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 324, in get_endpoint
    service_catalog = self.get_access(session).service_catalog
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 249, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 505, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
    resp = send(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
    raise exceptions.ConnectionRefused(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://mysite-ceilometer-6:5000/v2.0/tokens


The output from VM1:
---------------------------------

/mitaka-v2$ sudo tail -f /var/log/keystone/keystone-wsgi-public.log

<<<Successful case>>>>
2016-11-23 22:49:28.612 6830 INFO keystone.common.wsgi [req-da4329a5-0374-48de-9a07-229636042f64 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:49:28.713 6826 INFO keystone.common.wsgi [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:49:28.831 6826 INFO keystone.token.providers.fernet.utils [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
2016-11-23 22:49:28.865 6827 INFO keystone.common.wsgi [req-491da351-24c1-4b4d-9558-b6d66a280c35 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:49:28.895 6828 INFO keystone.common.wsgi [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:49:29.015 6828 INFO keystone.token.providers.fernet.utils [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/

<<<Unsuccessful case>>>>
2016-11-23 22:50:12.331 6829 INFO keystone.common.wsgi [req-f11122bc-bc3c-4521-81f3-d140bfce25b3 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:50:12.374 6830 INFO keystone.common.wsgi [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:50:12.481 6830 INFO keystone.token.providers.fernet.utils [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/


Thanks
Srikanth