<div dir="ltr"><div><div><div>I'd though this was happening before it got to a compute host since no host was listed in the instances table of the database, but I did manage to hunt down the RPC exception that happens on the compute node:<br>
<br>2013-08-13 12:12:09.055 AUDIT nova.compute.manager [req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5 6f9adccbd03e4d2186756896957a14bf] [instance: 62f92c7a-a7cf-41e7-83c5-055ad43ad78a] Starting instance...<br>
2013-08-13 12:12:11.279 ERROR nova.openstack.common.rpc.amqp [req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5 6f9adccbd03e4d2186756896957a14bf] Exception during message handling<br><br></div>and then a long trace, included below.<br>
<br></div>I suspected an incorrect rabbit password somewhere as the failure coincided with a config management update but I've mathc the passwords in config managenment and all teh openstack configs I can think of comparing both current values and backups from a known good state and haven't found that to be the case yet<br>
<br></div><div>Thanks,<br></div><div>-Jon<br><br></div>TRACE:<br><br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp temp_level, payload)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next()<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp pass<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next()<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next()<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1224, in run_instance<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp do_run_instance()<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1223, in do_run_instance<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 879, in _run_instance<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp self.gen.next()<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 800, in _run_instance<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image_meta = self._check_image_size(context, instance)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1006, in _check_image_size<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image_meta = _get_image_meta(context, instance['image_ref'])<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 268, in _get_image_meta<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return image_service.show(context, image_id)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 239, in show<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp _reraise_translated_image_exception(image_id)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 237, in show<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp image = self._client.call(context, 1, 'get', image_id)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 182, in call<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return getattr(client.images, method)(*args, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 104, in get<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp % urllib.quote(image_id))<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 260, in raw_request<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp return self._http_request(url, method, **kwargs)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 221, in _http_request<br>
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp raise exc.from_response(resp, body_str)<br>2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp ImageNotFound: Image 627b4902-f324-4615-a3bb-76f9fd22207a could not be found.<br>
<br><div><br><div><div><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 12, 2013 at 4:24 PM, John Bresnahan <span dir="ltr"><<a href="mailto:jbresnah@redhat.com" target="_blank">jbresnah@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 08/12/2013 09:09 AM, Jonathan Proulx wrote:<br>
><br>
> This is particularly odd since glance was working fine yesterday and to<br>
> my knowlege the only thing I did was turn on more compute nodes...<br>
><br>
><br>
> Now when I try and launch an instance it goes almost immediately to<br>
> error state with the fault:<br>
><br>
> {u'message': u'ImageNotFound', u'code': 404, u'details': u'Image<br>
> 627b4902-f324-4615-a3bb-76f9fd22207a could not be found.', u'created':<br>
> u'2013-08-12T18:53:46Z'}<br>
><br>
> I haven't yet found where the 404 is coming from, in debug mode the<br>
> glance-registry.log shows:<br>
><br>
> 2013-08-12 15:03:33.541 INFO glance.registry.api.v1.images<br>
> [73bb53ea-be14-496b-9405-098ff8f432aa 0be8fa0d641a4e778b9262bd2e5f40b5<br>
> 6f9adccbd03e4d2186756896957a14bf] Successfully retrieved image<br>
> 627b4902-f324-4615-a3bb-76f9fd22207a<br>
><br>
> which looks like success to me, also the python-glanceclient will<br>
> happily download the image.<br>
><br>
> Help?<br>
<br>
</div></div>Is there any thing in the nova compute log?<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote></div><br></div>