Open Stack

i would probably just try to get a token from vm2 and see if that's even 
possible. seems unrelated to ceilometerclient.

that said, i'm going to throw in obligatory: "ceilometer api is 
deprecated and has been unmaintained for over a year. i suggest you 
switch to another storage solution whether that's gnocchi or another 
time-series service"

On 23/11/16 06:35 PM, Srikanth Vavilapalli wrote:
> Hi
>
> We are facing few issues with keystone access in our setups.
>
> We have a setup which has three VMs. VM1 and VM3 are on same compute host whereas VM2 is on a different compute host.
>
> - VM1 (mysite-ceilometer-6): Running Mitaka Ceilometer service + Keystone (v2) service
> - VM2 (mysite-ceilometer-7): Running Ceilometer client
> - VM3 (mysite-ceilometer-8): Running Ceilometer client
>
> I have the keystone endpoints setup on VM1 as shown below: 'keystone endpoint-list'
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
> |              id   |   region  |              publicurl               |             internalurl              |                adminurl               |            service_id            |
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
> | 7c56f9e61 | RegionOne | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:35357/v2.0 | 4d871 |
> | adc3ac02f | RegionOne |   http://mysite-ceilometer-6:8777    |   http://mysite-ceilometer-6:8777    |    http://mysite-ceilometer-6:8777    | 50fb869 |
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
>
> I have set the following OS variables in both VM2 and VM3:
> OS_USERNAME="admin"
> OS_PASSWORD="password"
> OS_TENANT_NAME="admin"
> OS_AUTH_URL="http://mysite-ceilometer-6:5000/v2.0"
>
> The interesting observation I have made is that I am able to run the ceilometer or keystone commands successfully only from VM3 (which is on the same compute host as VM1) but not from VM2. Attached some of the logs I have collected during this observation. Can any of you help us with identifying what is going wrong here in this setup? Or any pointers on debugging why keystone/tokens REST API is not responding in some scenarios...? Thanks for ur help.
>
> The output from VM3 (mysite-ceilometer-8):
> ---------------------------------
> root at 69861113613c:/usr/local/share# ceilometer -d meter-list
> DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
> DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
> DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: req-da4329a5-0374-48de-9a07-229636042f64 Content-Type: application/json X-Distribution: Ubuntu
> RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
> DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
> DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
> DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
> DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: req-491da351-24c1-4b4d-9558-b6d66a280c35 Content-Type: application/json X-Distribution: Ubuntu
> RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
> DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
> DEBUG (client) REQ: curl -g -i -X 'GET' 'http://mysite-ceilometer-6:8777/v2/meters' -H 'User-Agent: ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}a9....'
> INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
> DEBUG (connectionpool) "GET /v2/meters HTTP/1.1" 200 2
> DEBUG (client) RESP: [200] {'Date': 'Wed, 23 Nov 2016 22:49:29 GMT', 'Content-Length': '2', 'Content-Type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-6bd80a0c-db08-4907-8b6b-4d477a6d7514', 'Server': 'Werkzeug/0.9.4 Python/2.7.6'}
>
> DEBUG (client) RESP BODY: []
>
> +------+------+------+-------------+---------+------------+
> | Name | Type | Unit | Resource ID | User ID | Project ID |
> +------+------+------+-------------+---------+------------+
> +------+------+------+-------------+---------+------------+
>
> The output from VM2 (mysite-ceilometer-7):
> ---------------------------------
>
> root at cadf877c4cdb:/usr/local/share# ceilometer -d meter-list
> DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
> DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
> DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:50:12 GMT x-openstack-request-id: req-f11122bc-bc3c-4521-81f3-d140bfce25b3 Content-Type: application/json X-Distribution: Ubuntu
> RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to http://mysite-ceilometer-6:5000/v2.0/tokens
> <<<<<Hangs here until time out>>>
> Traceback (most recent call last):
>   File "/usr/local/bin/ceilometer", line 11, in <module>
>     sys.exit(main())
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 292, in main
>     CeilometerShell().main(args)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 238, in main
>     client = ceiloclient.get_client(api_version, **client_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
>     return Client(version, endpoint, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
>     return client_class(*args, **client_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 67, in __init__
>     self.alarm_client = self._get_alarm_client(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 122, in _get_alarm_client
>     c.get("/")
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
>     return self.client_request("GET", url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
>     self, method, url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 244, in client_request
>     self.authenticate()
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 315, in authenticate
>     self.auth_plugin.authenticate(self)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/auth.py", line 203, in authenticate
>     self._do_authenticate(http_client)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 192, in _do_authenticate
>     endpoint = _get_endpoint(ks_session, **ks_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 162, in _get_endpoint
>     region_name=kwargs.get('region_name'))
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 664, in get_endpoint
>     return auth.get_endpoint(self, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 324, in get_endpoint
>     service_catalog = self.get_access(session).service_catalog
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 249, in get_access
>     self.auth_ref = self.get_auth_ref(session)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
>     authenticated=False, log=False)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 505, in post
>     return self.request(url, 'POST', **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
>     resp = send(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
>     raise exceptions.ConnectionRefused(msg)
> keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://mysite-ceilometer-6:5000/v2.0/tokens
>
>
> The output from VM1:
> ---------------------------------
>
> /mitaka-v2$ sudo tail -f /var/log/keystone/keystone-wsgi-public.log
>
> <<<Successful case>>>>
> 2016-11-23 22:49:28.612 6830 INFO keystone.common.wsgi [req-da4329a5-0374-48de-9a07-229636042f64 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:49:28.713 6826 INFO keystone.common.wsgi [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:49:28.831 6826 INFO keystone.token.providers.fernet.utils [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
> 2016-11-23 22:49:28.865 6827 INFO keystone.common.wsgi [req-491da351-24c1-4b4d-9558-b6d66a280c35 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:49:28.895 6828 INFO keystone.common.wsgi [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:49:29.015 6828 INFO keystone.token.providers.fernet.utils [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
>
> <<<Unsuccessful case>>>>
> 2016-11-23 22:50:12.331 6829 INFO keystone.common.wsgi [req-f11122bc-bc3c-4521-81f3-d140bfce25b3 - - - - -] GET http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:50:12.374 6830 INFO keystone.common.wsgi [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] POST http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:50:12.481 6830 INFO keystone.token.providers.fernet.utils [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
>
>
> Thanks
> Srikanth
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 
gord