[Openstack] essex rc1: rabbit time out when runing nova x509-get-root-cert
Pierre Amadio
pierre.amadio at canonical.com
Mon Mar 26 12:38:20 UTC 2012
Hi there !
I am trying to have a 4 node essex-rc1 setup running
nova/kesytone/glance (no swift, no horizon).
Things have been installed from the milestone-proposed branch of each
project.
I reach the stage where some nova command works (such as nova
flavor-list), but i cannot get a root certificate with
nova x509-get-root-cert
It looks to me nova-api send the request to rabbit, but that nobody
reply to it.
You will find more details (environment variables, nova-api logs, nova
--debug output and mostly empty rabbit log) in the attached pb-cert.txt
file.
Any idea what could be wrong ?
-------------- next part --------------
192.168.122.101: manager node (nova-api, nova-objectstore,nova-scheduler,nova-volume)
192.168.122.102: messaging (rabbit, keystone, glance, mysql)
192.168.122.103: nova-compute nova-network
192.168.122.104: nova-compute
nova command seems to work:
ubuntu at manager-node:~/creds$ cat openrc
# OS API settings
export NOVA_USERNAME=ubuntu
export NOVA_PROJECT_ID=ubuntu
export NOVA_PASSWORD=openstack
export NOVA_API_KEY=${NOVA_PASSWORD}
#where keystone runs
export NOVA_URL=http://192.168.122.102:35357/v2.0/
export NOVA_VERSION=1.1
export NOVA_REGION_NAME=RegionOne
export OS_AUTH_USER=${NOVA_USERNAME}
export OS_AUTH_KEY=${NOVA_PASSWORD}
export OS_AUTH_TENANT=${NOVA_PROJECT_ID}
export OS_AUTH_URL=${NOVA_URL}
export OS_AUTH_STRATEGY=keystone
# EC2 API settings
# Set the ec2 url so euca2ools works
export EC2_URL=${EC2_URL:-http://192.168.122.101:8773/services/Cloud}
# Access key is set in the initial keystone data to be the same as username
export EC2_ACCESS_KEY=3afad56c1e47481991e70b7094650d1a
# Secret key is set in the initial keystone data to the admin password
export EC2_SECRET_KEY=59cf599d36264764a7bd288ed87136bc
#export EC2_CERT=/home/ubuntu/creds/cacert.pem
#export EC2_PRIVATE_KEY=/home/ubuntu/creds/cert.pem
ubuntu at manager-node:~/creds$ . openrc
ubuntu at manager-node:~/creds$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor |
+----+-----------+-----------+------+-----------+------+-------+-------------+
| 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 |
| 2 | m1.small | 2048 | 10 | 20 | | 1 | 1.0 |
| 3 | m1.medium | 4096 | 10 | 40 | | 2 | 1.0 |
| 4 | m1.large | 8192 | 10 | 80 | | 4 | 1.0 |
| 5 | m1.xlarge | 16384 | 10 | 160 | | 8 | 1.0 |
+----+-----------+-----------+------+-----------+------+-------+-------------+
But i experience a time out when trying to launch the following:
ubuntu at manager-node:~/creds$ nova --debug x509-get-root-cert
connect: (192.168.122.102, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 192.168.122.102:35357\r\nContent-Length: 106\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "ubuntu", "passwordCredentials": {"username": "ubuntu", "password": "openstack"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 1645
header: Date: Mon, 26 Mar 2012 11:44:36 GMT
connect: (192.168.122.101, 8774)
send: u'GET /v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root HTTP/1.1\r\nHost: 192.168.122.101:8774\r\nx-auth-project-id: ubuntu\r\nx-auth-token: b59aedb8cca04336a47dec804a382657\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Content-Length: 128
header: Content-Type: application/json; charset=UTF-8
header: X-Compute-Request-Id: req-161614ff-cf70-4aba-a8f2-9116924eec7b
header: Date: Mon, 26 Mar 2012 11:45:36 GMT
DEBUG (shell:416) The server has either erred or is incapable of performing the requested operation. (HTTP 500)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/shell.py", line 413, in main
OpenStackComputeShell().main(sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/shell.py", line 364, in main
args.func(self.cs, args)
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/v1_1/shell.py", line 1458, in do_x509_get_root_cert
cacert = cs.certs.get()
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/v1_1/certs.py", line 48, in get
return self._get("/os-certificates/root", 'certificate')
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/base.py", line 149, in _get
resp, body = self.api.client.get(url)
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 136, in get
return self._cs_request(url, 'GET', **kwargs)
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 124, in _cs_request
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/python_novaclient-2012.1-py2.7.egg/novaclient/client.py", line 107, in request
raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
######################################################################################################################################"
Rabbit log
=INFO REPORT==== 26-Mar-2012::13:44:36 ===
starting TCP connection <0.479.0> from 192.168.122.101:41802
######################################################################################################################################
nova-api log:
2012-03-26 13:44:36 INFO nova.api.openstack.wsgi [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] GET http://192.168.122.101:8774/v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root
2012-03-26 13:44:36 DEBUG nova.api.openstack.wsgi [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Unrecognized Content-Type provided in request from (pid=14918) get_body /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py:697
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Making asynchronous call on cert ... from (pid=14918) multicall /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:321
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] MSG_ID is dce3eaea73bb4c6fb0e592c075b07562 from (pid=14918) multicall /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:324
2012-03-26 13:44:36 DEBUG nova.rpc.amqp [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Pool creating new connection from (pid=14918) create /usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py:59
2012-03-26 13:44:36 INFO nova.rpc.common [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Connected to AMQP server on 192.168.122.102:5672
2012-03-26 13:45:36 ERROR nova.rpc.common [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Timed out waiting for RPC response: timed out
(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 490, in ensure
(nova.rpc.common): TRACE: return method(*args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 567, in _consume
(nova.rpc.common): TRACE: return self.connection.drain_events(timeout=timeout)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events
(nova.rpc.common): TRACE: return self.transport.drain_events(self.connection, **kwargs)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events
(nova.rpc.common): TRACE: return connection.drain_events(**kwargs)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events
(nova.rpc.common): TRACE: return self.wait_multi(self.channels.values(), timeout=timeout)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi
(nova.rpc.common): TRACE: chanmap.keys(), allowed_methods, timeout=timeout)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple
(nova.rpc.common): TRACE: channel, method_sig, args, content = self.read_timeout(timeout)
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout
(nova.rpc.common): TRACE: return self.method_reader.read_method()
(nova.rpc.common): TRACE: File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
(nova.rpc.common): TRACE: raise m
(nova.rpc.common): TRACE: timeout: timed out
(nova.rpc.common): TRACE:
2012-03-26 13:45:36 ERROR nova.api.openstack [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] Caught error: Timeout while waiting on RPC response.
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/__init__.py", line 41, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/keystone-2012.1-py2.7.egg/keystone/middleware/auth_token.py", line 176, in __call__
(nova.api.openstack): TRACE: return self.app(env, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 803, in __call__
(nova.api.openstack): TRACE: content_type, body, accept)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 851, in _process_stack
(nova.api.openstack): TRACE: action_result = self.dispatch(meth, request, action_args)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/wsgi.py", line 926, in dispatch
(nova.api.openstack): TRACE: return method(req=request, **action_args)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/api/openstack/compute/contrib/certificates.py", line 79, in show
(nova.api.openstack): TRACE: "args": {"project_id": context.project_id}})
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/__init__.py", line 68, in call
(nova.api.openstack): TRACE: return _get_impl().call(context, topic, msg, timeout)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 674, in call
(nova.api.openstack): TRACE: return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py", line 338, in call
(nova.api.openstack): TRACE: rv = list(rv)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/amqp.py", line 299, in __iter__
(nova.api.openstack): TRACE: self._iterator.next()
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 572, in iterconsume
(nova.api.openstack): TRACE: yield self.ensure(_error_callback, _consume)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 503, in ensure
(nova.api.openstack): TRACE: error_callback(e)
(nova.api.openstack): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova-2012.1-py2.7.egg/nova/rpc/impl_kombu.py", line 553, in _error_callback
(nova.api.openstack): TRACE: raise rpc_common.Timeout()
(nova.api.openstack): TRACE: Timeout: Timeout while waiting on RPC response.
(nova.api.openstack): TRACE:
2012-03-26 13:45:36 INFO nova.api.openstack [req-161614ff-cf70-4aba-a8f2-9116924eec7b 8e5902533835427b8f812ec60c909546 a07fd73258684caaa86edcaabd7a16b0] http://192.168.122.101:8774/v1.1/a07fd73258684caaa86edcaabd7a16b0/os-certificates/root returned with HTTP 500
More information about the Openstack
mailing list