Open Stack

I'd though this was happening before it got to a compute host since no host
was listed in the instances table of the database, but I did manage to hunt
down the RPC exception that happens on the compute node:

2013-08-13 12:12:09.055 AUDIT nova.compute.manager
[req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5
6f9adccbd03e4d2186756896957a14bf] [instance:
62f92c7a-a7cf-41e7-83c5-055ad43ad78a] Starting instance...
2013-08-13 12:12:11.279 ERROR nova.openstack.common.rpc.amqp
[req-3ec8ea52-2ef7-4a38-8ccf-4d629a2a16df 0be8fa0d641a4e778b9262bd2e5f40b5
6f9adccbd03e4d2186756896957a14bf] Exception during message handling

and then a long trace, included below.

I suspected an incorrect rabbit password somewhere as the failure coincided
with a config management update but I've mathc the passwords in config
managenment and all teh openstack configs I can think of comparing both
current values and backups from a known good state and haven't found that
to be the case yet

Thanks,
-Jon

TRACE:

2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
430, in _process_data
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     rval
= self.proxy.dispatch(ctxt, version, method, **args)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 133, in dispatch
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return f(self, context, *args, **kw)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
decorated_function
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     pass
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
decorated_function
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
decorated_function
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
decorated_function
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     e,
sys.exc_info())
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
decorated_function
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1224, in
run_instance
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
do_run_instance()
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
242, in inner
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
retval = f(*args, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1223, in
do_run_instance
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
admin_password, is_first_time, node, instance)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 879, in
_run_instance
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
self._set_instance_error_state(context, instance['uuid'])
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 800, in
_run_instance
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
image_meta = self._check_image_size(context, instance)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1006, in
_check_image_size
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
image_meta = _get_image_meta(context, instance['image_ref'])
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 268, in
_get_image_meta
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return image_service.show(context, image_id)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 239, in show
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
_reraise_translated_image_exception(image_id)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 237, in show
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     image
= self._client.call(context, 1, 'get', image_id)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 182, in call
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return getattr(client.images, method)(*args, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 104, in
get
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     %
urllib.quote(image_id))
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 260,
in raw_request
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
return self._http_request(url, method, **kwargs)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 221,
in _http_request
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp     raise
exc.from_response(resp, body_str)
2013-08-13 12:28:41.317 9593 TRACE nova.openstack.common.rpc.amqp
ImageNotFound: Image 627b4902-f324-4615-a3bb-76f9fd22207a could not be
found.





On Mon, Aug 12, 2013 at 4:24 PM, John Bresnahan <jbresnah at redhat.com> wrote:

> On 08/12/2013 09:09 AM, Jonathan Proulx wrote:
> >
> > This is particularly odd since glance was working fine yesterday and to
> > my knowlege the only thing I did was turn on more compute nodes...
> >
> >
> > Now when I try and launch an instance it goes almost immediately to
> > error state with the fault:
> >
> > {u'message': u'ImageNotFound', u'code': 404, u'details': u'Image
> > 627b4902-f324-4615-a3bb-76f9fd22207a could not be found.', u'created':
> > u'2013-08-12T18:53:46Z'}
> >
> > I haven't yet found where the 404 is coming from, in debug mode the
> > glance-registry.log shows:
> >
> > 2013-08-12 15:03:33.541 INFO glance.registry.api.v1.images
> > [73bb53ea-be14-496b-9405-098ff8f432aa 0be8fa0d641a4e778b9262bd2e5f40b5
> > 6f9adccbd03e4d2186756896957a14bf] Successfully retrieved image
> > 627b4902-f324-4615-a3bb-76f9fd22207a
> >
> > which looks like success to me, also the python-glanceclient will
> > happily download the image.
> >
> > Help?
>
> Is there any thing in the nova compute log?
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130813/97c76884/attachment.html>