Open Stack

glanceclient has always been that way.... it's terrible. :(

the nova message timeout looks like it's becauce nova-cert is not
running, and it keeps trying to ping it. What actions were you doing
that triggered that path?

On 04/16/2015 07:23 PM, Danny Choi (dannchoi) wrote:
> When use devstack with Kilo, after stack.sh, the following tracebacks are logged:
> 
> localadmin at qa4:/opt/stack/logs$ grep -r Traceback *
> g-api.log:2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log:2015-04-16 14:16:20.237 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log:2015-04-16 14:16:21.973 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log:2015-04-16 14:16:24.538 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log.2015-04-16-141127:2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log.2015-04-16-141127:2015-04-16 14:16:20.237 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log.2015-04-16-141127:2015-04-16 14:16:21.973 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> g-api.log.2015-04-16-141127:2015-04-16 14:16:24.538 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> n-api.log:2015-04-16 14:18:28.163 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log:2015-04-16 14:19:34.470 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log:2015-04-16 14:20:39.624 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log:2015-04-16 14:21:44.879 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log:2015-04-16 14:22:49.676 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log:2015-04-16 14:23:55.475 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:18:28.163 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:19:34.470 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:20:39.624 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:21:44.879 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:22:49.676 TRACE nova.api.openstack Traceback (most recent call last):
> n-api.log.2015-04-16-141127:2015-04-16 14:23:55.475 TRACE nova.api.openstack Traceback (most recent call last):
> 
> Traceback #1:
> ============
> 
> 2015-04-16 14:16:17.996 ERROR glance.registry.client.v1.client [req-dc54f751-eb4b-4a03-9146-52623ed733a9 d0feb86a66d54df8b4aff1848c35729e 8afd7feb78ee445b8d642a66c96d49d5] 
> Registry client request GET /images/cirros-0.3.3-x86_64-uec-kernel raised NotFound
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client Traceback (most recent call last):
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client   File "/opt/stack/glance/glance/registry/client/v1/client.py", line 117, in do_request
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client     **kwargs)
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client   File "/opt/stack/glance/glance/common/client.py", line 71, in wrapped
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client     return func(self, *args, **kwargs)
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client   File "/opt/stack/glance/glance/common/client.py", line 376, in do_request
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client     headers=copy.deepcopy(headers))
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client   File "/opt/stack/glance/glance/common/client.py", line 88, in wrapped
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client     return func(self, method, url, body, headers)
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client   File "/opt/stack/glance/glance/common/client.py", line 523, in _do_request
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client     raise exception.NotFound(res.read())
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client NotFound: 404 Not Found
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client 
> 2015-04-16 14:16:17.996 TRACE glance.registry.client.v1.client The resource could not be found.
> 
> Traceback #2:
> ==============
> 
> 2015-04-16 14:17:28.159 INFO oslo_messaging._drivers.impl_rabbit [req-78c0070a-2a89-452f-8f3b-7aa549b9a3ed admin admin] Connected to AMQP server on 172.29.172.1
> 61:5672
> 2015-04-16 14:18:28.163 ERROR nova.api.openstack [req-78c0070a-2a89-452f-8f3b-7aa549b9a3ed admin admin] Caught error: Timed out waiting for a reply to message I
> D 79c4306de3fc46aa986664735573100b
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack Traceback (most recent call last):
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return req.get_response(self.application)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     application, catch_exc_info=False)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return resp(environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return self._call_app(env, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return resp(environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return resp(environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     response = self.app(environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return resp(environ, start_response)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in __call__
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     content_type, body, accept)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return method(req=request, **action_args)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/contrib/certificates.py", line 48, in show
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     project_id=context.project_id)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/opt/stack/nova/nova/cert/rpcapi.py", line 93, in fetch_ca
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     return cctxt.call(ctxt, 'fetch_ca', project_id=project_id)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     retry=self.retry)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     timeout=timeout, retry=retry)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     retry=retry)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     result = self._waiter.wait(msg_id, timeout)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     message = self.waiters.get(msg_id, timeout=timeout)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack     'to message ID %s' % msg_id)
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 79c4306de3fc46aa986664735573100b
> 2015-04-16 14:18:28.163 TRACE nova.api.openstack 
> 
> #####################
> 
> stack.sh does completed. 
> 
> Are these tracebacks harmful?
> 
> Also, I noticed "nova-cert" and "nova-consoleauth" are missing from show output:
> 
> localadmin at qa4:~/devstack-kilo$ nova service-list
> +----+----------------+------+----------+---------+-------+----------------------------+-----------------+
> | Id | Binary         | Host | Zone     | Status  | State | Updated_at                 | Disabled Reason |
> +----+----------------+------+----------+---------+-------+----------------------------+-----------------+
> | 1  | nova-conductor | qa4  | internal | enabled | up    | 2015-04-16T18:51:40.000000 | -               |
> | 2  | nova-scheduler | qa4  | internal | enabled | up    | 2015-04-16T18:51:44.000000 | -               |
> | 3  | nova-compute   | qa5  | nova     | enabled | up    | 2015-04-16T18:51:39.000000 | -               |
> | 4  | nova-compute   | qa6  | nova     | enabled | up    | 2015-04-16T18:51:46.000000 | -               |
> +----+----------------+------+----------+---------+-------+----------------------------+-----------------+
> 
> And I cannot launch the VM's console, both from Horizon and CLI.
> 
> Note that I’m using the same local.conf settings that are working fine in Juno.
> 
> 
> Thanks,
> 
> Danny
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net