Open Stack

Hi Gordon

Yes, I agree, this issue is not related to ceilometer. We have verified that none of the keystone commands (keystone endpoint-list, keystone catalog, keystone user-list) are working in that VM2, they all stuck in getting the token-get operation (/v2.0/tokens).

If I had to suspect the keystone server setup in my VM1, then the keystone client on VM3 also wouldn't work. So seems something else is going wrong here. The keystone debugs logs also not helping much here...

On the last part, could you plz point us to any existing wiki pages that describe to migrate away from ceilometer-api? Thanks for ur help...

Thanks
Srikanth

-----Original Message-----
From: gordon chung [mailto:gord at live.ca] 
Sent: Wednesday, November 23, 2016 4:20 PM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [keystone][ceilometer] 'ceilometer -d meter-list' hangs in "Making authentication request to keystone"

i would probably just try to get a token from vm2 and see if that's even possible. seems unrelated to ceilometerclient.

that said, i'm going to throw in obligatory: "ceilometer api is deprecated and has been unmaintained for over a year. i suggest you switch to another storage solution whether that's gnocchi or another time-series service"

On 23/11/16 06:35 PM, Srikanth Vavilapalli wrote:
> Hi
>
> We are facing few issues with keystone access in our setups.
>
> We have a setup which has three VMs. VM1 and VM3 are on same compute host whereas VM2 is on a different compute host.
>
> - VM1 (mysite-ceilometer-6): Running Mitaka Ceilometer service + 
> Keystone (v2) service
> - VM2 (mysite-ceilometer-7): Running Ceilometer client
> - VM3 (mysite-ceilometer-8): Running Ceilometer client
>
> I have the keystone endpoints setup on VM1 as shown below: 'keystone endpoint-list'
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
> |              id   |   region  |              publicurl               |             internalurl              |                adminurl               |            service_id            |
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
> | 7c56f9e61 | RegionOne | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:35357/v2.0 | 4d871 |
> | adc3ac02f | RegionOne |   http://mysite-ceilometer-6:8777    |   http://mysite-ceilometer-6:8777    |    http://mysite-ceilometer-6:8777    | 50fb869 |
> +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
>
> I have set the following OS variables in both VM2 and VM3:
> OS_USERNAME="admin"
> OS_PASSWORD="password"
> OS_TENANT_NAME="admin"
> OS_AUTH_URL="http://mysite-ceilometer-6:5000/v2.0"
>
> The interesting observation I have made is that I am able to run the ceilometer or keystone commands successfully only from VM3 (which is on the same compute host as VM1) but not from VM2. Attached some of the logs I have collected during this observation. Can any of you help us with identifying what is going wrong here in this setup? Or any pointers on debugging why keystone/tokens REST API is not responding in some scenarios...? Thanks for ur help.
>
> The output from VM3 (mysite-ceilometer-8):
> ---------------------------------
> root at 69861113613c:/usr/local/share# ceilometer -d meter-list DEBUG 
> (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): 
> mysite-ceilometer-6 DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 
> 345 DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token 
> Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) 
> Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT 
> x-openstack-request-id: req-da4329a5-0374-48de-9a07-229636042f64 
> Content-Type: application/json X-Distribution: Ubuntu RESP BODY: 
> {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", 
> "media-types": [{"base": "application/json", "type": 
> "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", 
> "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": 
> "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", 
> "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293 DEBUG 
> (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): 
> mysite-ceilometer-6 DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 
> 345 DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token 
> Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) 
> Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT 
> x-openstack-request-id: req-491da351-24c1-4b4d-9558-b6d66a280c35 
> Content-Type: application/json X-Distribution: Ubuntu RESP BODY: 
> {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", 
> "media-types": [{"base": "application/json", "type": 
> "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", 
> "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": 
> "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", 
> "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293 DEBUG 
> (client) REQ: curl -g -i -X 'GET' 'http://mysite-ceilometer-6:8777/v2/meters' -H 'User-Agent: ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}a9....'
> INFO (connectionpool) Starting new HTTP connection (1): 
> mysite-ceilometer-6 DEBUG (connectionpool) "GET /v2/meters HTTP/1.1" 
> 200 2 DEBUG (client) RESP: [200] {'Date': 'Wed, 23 Nov 2016 22:49:29 
> GMT', 'Content-Length': '2', 'Content-Type': 'application/json; 
> charset=UTF-8', 'x-openstack-request-id': 
> 'req-6bd80a0c-db08-4907-8b6b-4d477a6d7514', 'Server': 'Werkzeug/0.9.4 
> Python/2.7.6'}
>
> DEBUG (client) RESP BODY: []
>
> +------+------+------+-------------+---------+------------+
> | Name | Type | Unit | Resource ID | User ID | Project ID |
> +------+------+------+-------------+---------+------------+
> +------+------+------+-------------+---------+------------+
>
> The output from VM2 (mysite-ceilometer-7):
> ---------------------------------
>
> root at cadf877c4cdb:/usr/local/share# ceilometer -d meter-list DEBUG 
> (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
> INFO (connectionpool) Starting new HTTP connection (1): 
> mysite-ceilometer-6 DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 
> 345 DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token 
> Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) 
> Connection: Keep-Alive Date: Wed, 23 Nov 2016 22:50:12 GMT 
> x-openstack-request-id: req-f11122bc-bc3c-4521-81f3-d140bfce25b3 
> Content-Type: application/json X-Distribution: Ubuntu RESP BODY: 
> {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", 
> "media-types": [{"base": "application/json", "type": 
> "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", 
> "links": [{"href": "http://mysite-ceilometer-6:5000/v2.0/", "rel": 
> "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", 
> "rel": "describedby"}]}}
>
> DEBUG (v2) Making authentication request to 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> <<<<<Hangs here until time out>>>
> Traceback (most recent call last):
>   File "/usr/local/bin/ceilometer", line 11, in <module>
>     sys.exit(main())
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 292, in main
>     CeilometerShell().main(args)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line 238, in main
>     client = ceiloclient.get_client(api_version, **client_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
>     return Client(version, endpoint, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
>     return client_class(*args, **client_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 67, in __init__
>     self.alarm_client = self._get_alarm_client(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 122, in _get_alarm_client
>     c.get("/")
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
>     return self.client_request("GET", url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
>     self, method, url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 244, in client_request
>     self.authenticate()
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 315, in authenticate
>     self.auth_plugin.authenticate(self)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/auth.py", line 203, in authenticate
>     self._do_authenticate(http_client)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 192, in _do_authenticate
>     endpoint = _get_endpoint(ks_session, **ks_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 162, in _get_endpoint
>     region_name=kwargs.get('region_name'))
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 664, in get_endpoint
>     return auth.get_endpoint(self, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 324, in get_endpoint
>     service_catalog = self.get_access(session).service_catalog
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 249, in get_access
>     self.auth_ref = self.get_auth_ref(session)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
>     authenticated=False, log=False)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 505, in post
>     return self.request(url, 'POST', **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request
>     resp = send(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request
>     raise exceptions.ConnectionRefused(msg)
> keystoneauth1.exceptions.connection.ConnectFailure: Unable to 
> establish connection to http://mysite-ceilometer-6:5000/v2.0/tokens
>
>
> The output from VM1:
> ---------------------------------
>
> /mitaka-v2$ sudo tail -f /var/log/keystone/keystone-wsgi-public.log
>
> <<<Successful case>>>>
> 2016-11-23 22:49:28.612 6830 INFO keystone.common.wsgi 
> [req-da4329a5-0374-48de-9a07-229636042f64 - - - - -] GET 
> http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:49:28.713 6826 INFO keystone.common.wsgi 
> [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] POST 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:49:28.831 6826 INFO 
> keystone.token.providers.fernet.utils 
> [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] Loaded 2 
> encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
> 2016-11-23 22:49:28.865 6827 INFO keystone.common.wsgi 
> [req-491da351-24c1-4b4d-9558-b6d66a280c35 - - - - -] GET 
> http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:49:28.895 6828 INFO keystone.common.wsgi 
> [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] POST 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:49:29.015 6828 INFO 
> keystone.token.providers.fernet.utils 
> [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] Loaded 2 
> encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
>
> <<<Unsuccessful case>>>>
> 2016-11-23 22:50:12.331 6829 INFO keystone.common.wsgi 
> [req-f11122bc-bc3c-4521-81f3-d140bfce25b3 - - - - -] GET 
> http://mysite-ceilometer-6:5000/v2.0/
> 2016-11-23 22:50:12.374 6830 INFO keystone.common.wsgi 
> [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] POST 
> http://mysite-ceilometer-6:5000/v2.0/tokens
> 2016-11-23 22:50:12.481 6830 INFO 
> keystone.token.providers.fernet.utils 
> [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] Loaded 2 
> encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys/
>
>
> Thanks
> Srikanth
>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

--
gord

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev