Open Stack

Hey guys, we finally got MultiZone working on Essex with KVM as 
hypervisor (trunk version 2012.1-dev (2012.1-LOCALBRANCH:LOCALREVISION)) 
we have one Parent zone and one child zone (named "zoneE") all 
integrated with keystone ( nova, glance ) and we are able to spawn 
instances across both zones !
After making this work after a couple of days of work, configuration and 
debbuging we have one doubt, and one ERROR.
So, lets hope you can help us on both of them.

THE DOUBT:
This is deffinitely because we dont clearly understand the whole concept 
but, why is that "nova.scheduler.least_cost.weighted_sum" function 
chooses as prefered host the LESS RAM memory host ? this seems to choose 
just one host till is fully depleted, and then, chooses the other one ! 
that doesnt sound like a good way of balancing instances across the 
zones by default, so, since we didnt know how to fix that behavior, just 
added a "reverse=True" in the sorted functions, so problem "fixed".
The thing is, if there is (maybe trough nova.conf) a way of controlling 
this ( the sort order ), can any one shed some light on how to do it ?

THE ERROR / BUG:
At the moment that you spawn successfully an instance from the parent 
zone, to the child zone (the instance is successfully spawned and 
running on the child zone) the "nova list" (or curl to server/detail) to 
the parent zone just BROKE.
It apperars that when the parent zone calls the "server/detail" of the 
child zone, the json returned by the child, has the "uuid" field printed 
into the "id" field, so you can see on nova-api.log, having a keyerror 
"uuid" that breaks the call, you can check the pasteBin link below, or 
you can keep reading and watch the nova-api.log stack from here.

Waiting for some feedback !
Cheers!
Alejandro.

PASTEBIN LINK:
--------------
http://pastebin.com/VGt8A3HX

ENTIRE NOVA-API STACK TRACE:
----------------------------
2012-01-25 14:42:15,793 DEBUG nova [-] HTTP PERF: 0.10457 seconds to GET 
172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) 
from (pid=7488) getresponse 
/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
2012-01-25 14:42:15,894 DEBUG nova [-] HTTP PERF: 0.10093 seconds to GET 
172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) 
from (pid=7488) getresponse 
/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
2012-01-25 14:42:15,897 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Matched GET 
/9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) 
__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
2012-01-25 14:42:15,897 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Route path: 
'/:(project_id)/servers/:(id)', defaults: {'action': u'process', 
'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource 
object at 0x261b850>} from (pid=7488) __call__ 
/usr/lib/pymodules/python2.6/routes/middleware.py:102
2012-01-25 14:42:15,898 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'process', 
'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource 
object at 0x261b850>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5', 
'id': u'detail'} from (pid=7488) __call__ 
/usr/lib/pymodules/python2.6/routes/middleware.py:103
2012-01-25 14:42:15,899 INFO nova.api.openstack.wsgi 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] GET 
http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
2012-01-25 14:42:15,899 DEBUG nova.api.openstack.wsgi 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in 
request from (pid=7488) deserialize_body 
/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
2012-01-25 14:42:15,900 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Matched GET 
/9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) 
__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
2012-01-25 14:42:15,901 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Route path: 
'/{project_id}/servers/detail', defaults: {'action': u'detail', 
'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>} 
from (pid=7488) __call__ 
/usr/lib/pymodules/python2.6/routes/middleware.py:102
2012-01-25 14:42:15,902 DEBUG routes.middleware 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'detail', 
'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>, 
'project_id': u'9574193127d1439fb09d1c0ef3eb26d5'} from (pid=7488) 
__call__ /usr/lib/pymodules/python2.6/routes/middleware.py:103
2012-01-25 14:42:15,902 INFO nova.api.openstack.wsgi 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] GET 
http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
2012-01-25 14:42:15,903 DEBUG nova.api.openstack.wsgi 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in 
request from (pid=7488) deserialize_body 
/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
2012-01-25 14:42:15,903 DEBUG nova.compute.api 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Searching by: {'deleted': False, 
'local_zone_only': False} from (pid=7488) get_all 
/usr/lib/python2.6/dist-packages/nova/compute/api.py:937
2012-01-25 14:42:16,116 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i 
http://172.16.146.139:5000/v2.0/tokens -X POST -H "Content-Type: 
application/json" -H "User-Agent: python-novaclient"
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,116 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] REQ BODY: {"auth": 
{"passwordCredentials": {"username": "esmadmin", "password": "123456"}}}
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:78
2012-01-25 14:42:16,117 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] RESP:{'date': 'Wed, 25 Jan 2012 
19:42:16 GMT', 'vary': 'X-Auth-Token', 'content-length': '1583', 
'status': '200', 'content-type': 'application/json; charset=UTF-8'} 
{"access": {"token": {"expires": "2012-01-26T15:42:15.970710", 
"tenants": [{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": 
"esmadmin"}], "id": "d9164294-0e8f-4f01-94c6-eb4b460c95ee", "tenant": 
{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}}, 
"serviceCatalog": [{"endpoints": [{"adminURL": 
"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", 
"region": "zoneE", "id": "5", "internalURL": 
"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", 
"publicURL": 
"http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], 
"type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": 
"http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "region": 
"RegionOne", "id": "2", "internalURL": 
"http://172.16.147.123:9292/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", 
"publicURL": 
"http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type": 
"image", "name": "glance"}, {"endpoints": [{"adminURL": 
"http://172.16.146.139:35357/v2.0", "region": "RegionOne", "id": "3", 
"internalURL": "http://172.16.146.139:5000/v2.0", "publicURL": 
"http://172.16.146.139:5000/v2.0"}], "type": "identity", "name": 
"keystone"}, {"endpoints": [{"adminURL": "http://172.16.0.88:8080/", 
"region": "RegionOne", "id": "4", "internalURL": 
"http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b", 
"publicURL": 
"http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b"}], 
"type": "object-store", "name": "swift"}], "user": {"id": 
"4a4774f4374b4a928df9319009302a34", "roles": [{"name": "KeystoneAdmin", 
"id": "3"}], "name": "esmadmin"}}}
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,119 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Using Endpoint URL: 
http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints 
from (pid=7488) _fetch_endpoints_from_auth 
/usr/lib/python2.6/dist-packages/novaclient/client.py:185
2012-01-25 14:42:16,238 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i 
http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints 
-X GET -H "X-Auth_Token: d9164294-0e8f-4f01-94c6-eb4b460c95ee" -H 
"User-Agent: python-novaclient"
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,239 DEBUG novaclient.client 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] RESP:{'status': '200', 
'content-length': '870', 'content-location': 
'http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints', 
'vary': 'X-Auth-Token', 'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 
'content-type': 'application/json; charset=UTF-8'} {"endpoints_links": 
[{"href": 
"http://172.16.146.139:35357/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints?marker=3&limit=10", 
"rel": "next"}], "endpoints": [{"internalURL": 
"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", 
"name": "nova", "adminURL": 
"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", 
"region": "RegionOne", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", 
"type": "compute", "id": 2, "publicURL": 
"http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}, 
{"internalURL": 
"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", 
"name": "nova", "adminURL": 
"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", 
"region": "zoneE", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", 
"type": "compute", "id": 3, "publicURL": 
"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}]}
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,737 DEBUG novaclient.client [-] REQ: curl -i 
http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail 
-X GET -H "User-Agent: python-novaclient" -H "X-Auth-Token: 
8de5e217-9615-443c-8b7e-09cabf4b4af8"
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,738 DEBUG novaclient.client [-] RESP:{'date': 'Wed, 
25 Jan 2012 19:42:16 GMT', 'status': '200', 'content-length': '2397', 
'content-type': 'application/json', 'content-location': 
u'http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail'} 
{"servers": [{"OS-EXT-STS:task_state": null, "addresses": {}, "links": 
[{"href": 
"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", 
"rel": "self"}, {"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", 
"rel": "bookmark"}], "image": {"id": 
"b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", 
"rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": 
"1", "links": [{"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", 
"rel": "bookmark"}]}, "id": "d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", 
"user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", 
"accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, 
"config_drive": "", "status": "ACTIVE", "updated": 
"2012-01-25T19:41:15Z", "hostId": 
"64fe9962f6dc6c376f4e8f9eb6c871aaf0da3771d9a415f0bf60a770", "key_name": 
"", "name": "vmtest1", "created": "2012-01-25T19:40:57Z", "tenant_id": 
"9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": 
"vmtest1"}}, {"OS-EXT-STS:task_state": null, "addresses": {}, "links": 
[{"href": 
"http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", 
"rel": "self"}, {"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", 
"rel": "bookmark"}], "image": {"id": 
"b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", 
"rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": 
"1", "links": [{"href": 
"http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", 
"rel": "bookmark"}]}, "id": "8f00967a-bb2a-4f24-bf5d-48b1f3c77388", 
"user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", 
"accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, 
"config_drive": "", "status": "ACTIVE", "updated": 
"2012-01-25T19:38:31Z", "hostId": 
"028036137ae671554f81f2262ee84138a1f375d32b91550a34fdaca1", "key_name": 
"", "name": "vmtest1", "created": "2012-01-25T19:38:22Z", "tenant_id": 
"9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": 
"vmtest1"}}]}
  from (pid=7488) http_log 
/usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,741 ERROR nova.api.openstack.v2 
[2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 
9574193127d1439fb09d1c0ef3eb26d5] Caught error: 'uuid'
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/__init__.py", 
line 62, in __call__
(nova.api.openstack.v2): TRACE:     return 
req.get_response(self.application)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in 
call_application
(nova.api.openstack.v2): TRACE:     app_iter = application(self.environ, 
start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", 
line 186, in __call__
(nova.api.openstack.v2): TRACE:     return self._forward_request(env, 
start_response, proxy_headers)
(nova.api.openstack.v2): TRACE:   File 
"/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", 
line 312, in _forward_request
(nova.api.openstack.v2): TRACE:     return self.app(env, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE:     resp = self.call_func(req, *args, 
**self.kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 499, 
in __call__
(nova.api.openstack.v2): TRACE:     response = 
req.get_response(self.application)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in 
call_application
(nova.api.openstack.v2): TRACE:     app_iter = application(self.environ, 
start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack.v2): TRACE:     response = self.app(environ, 
start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE:     resp = self.call_func(req, *args, 
**self.kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, 
in __call__
(nova.api.openstack.v2): TRACE:     action_result = 
self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, 
in dispatch
(nova.api.openstack.v2): TRACE:     return 
controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/extensions.py", 
line 166, in process
(nova.api.openstack.v2): TRACE:     res = req.get_response(self.application)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE:     application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in 
call_application
(nova.api.openstack.v2): TRACE:     app_iter = application(self.environ, 
start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack.v2): TRACE:     response = self.app(environ, 
start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE:     return resp(environ, start_response)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE:     resp = self.call_func(req, *args, 
**self.kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE:     return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, 
in __call__
(nova.api.openstack.v2): TRACE:     action_result = 
self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, 
in dispatch
(nova.api.openstack.v2): TRACE:     return 
controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", 
line 75, in detail
(nova.api.openstack.v2): TRACE:     servers = self._get_servers(req, 
is_detail=True)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", 
line 150, in _get_servers
(nova.api.openstack.v2): TRACE:     self._add_instance_faults(context, 
limited_list)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", 
line 89, in _add_instance_faults
(nova.api.openstack.v2): TRACE:     faults = 
self.compute_api.get_instance_faults(ctxt, instances)
(nova.api.openstack.v2): TRACE:   File 
"/usr/lib/python2.6/dist-packages/nova/compute/api.py", line 1607, in 
get_instance_faults
(nova.api.openstack.v2): TRACE:     uuids = [instance['uuid'] for 
instance in instances]
(nova.api.openstack.v2): TRACE: KeyError: 'uuid'
(nova.api.openstack.v2): TRACE: