<div dir="ltr">I was finally able to determine the cause of this problem. A bug report with full description has been submitted here:<div><a href="https://bugs.launchpad.net/glance/+bug/1198566">https://bugs.launchpad.net/glance/+bug/1198566</a></div>
<div><br></div><div style>best,</div><div style>Mark</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 27, 2013 at 7:57 PM, Mark A. Nye <span dir="ltr"><<a href="mailto:marknye@clubofhumanbeings.com" target="_blank">marknye@clubofhumanbeings.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div>Hello,</div><div><br></div><div>We have a working two host Folsom (<span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px">2012.2.3 on Ubuntu 12.04</span>) OpenStack cluster with a controller and compute node. This afternoon we added a second compute node, but attempts to spawn instances on the new node fail with a glance client exception (see example log below).</div>
<div><br></div><div>I've triple-checked our new nova.conf and api-paste.ini files, which are identical to what we have on the working compute node, except for the local IP address change to metadata_host, vncserver_proxyclient_address, and my_ip local.</div>
<div><br></div><div>When I run "nova-manage service list" the nova-compute and nova-network services on all three machines report as status :-).</div><div><br></div><div>The ONLY significant difference I'm seeing is that the new compute node is running the 2012.2.4 version of nova-network, nova-compute, and nova-api-metadata. I wouldn't expect this to be a problem, but maybe a bug or incompatibility was introduced in 2012.2.4? I'd like to try rolling back to 2012.2.3, but I can't find a copy of the older .deb packages.</div>
<div><br></div><div>Am I missing something obvious? Can anyone offer a suggestion?</div><div><br></div><div>best,<br>Mark</div><div><br></div><div><br></div><div><br></div><div>
<br></div><div><br></div><div><div>2013-06-28 02:12:47 5290 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp temp_level, payload)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in decorated_function</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp pass</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in decorated_function</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in decorated_function</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in decorated_function</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in run_instance</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp do_run_instance()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in do_run_instance</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in _run_instance</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in _run_instance</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp is_first_time, request_spec, filter_properties)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 503, in _run_instance</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp injected_files, admin_password)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in _spawn</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp block_device_info)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp temp_level, payload)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp self.gen.next()</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099, in spawn</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp admin_pass=admin_password)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365, in _create_image</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp project_id=instance['project_id'])</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 131, in cache</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp *args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 178, in create_image</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp prepare_template(target=base, *args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 122, in call_if_not_exists</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp fetch_func(target=target, *args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp images.fetch_to_raw(context, image_id, target, user_id, project_id)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp fetch(context, image_href, path_tmp, user_id, project_id)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp image_service.download(context, image_id, image_file)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in download</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp _reraise_translated_image_exception(image_id)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in download</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp image_chunks = self._client.call(context, 1, 'data', image_id)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return getattr(client.images, method)(*args, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in data</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp resp, body = self.api.raw_request('GET', '/v1/images/%s' % image_id)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191, in raw_request</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp return self._http_request(url, method, **kwargs)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158, in _http_request</div><div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp raise exc.from_response(resp)</div>
<div>2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp HTTPInternalServerError: HTTPInternalServerError (HTTP 500)</div></div></div>
</blockquote></div><br></div>