Open Stack

Hello Jesse,

I tried turning SSL on quantum and I am running into a problem. I have a compute node with nova running on it and everything else running on a controller node. When I change quantum to use its wsgi interface, I am getting an error from the quantum-server.log file:


Ø  [Wed Nov 27 14:08:56 2013] [debug] ssl_engine_kernel.c(1879): OpenSSL: Read: SSLv3 read client certificate A

Ø  [Wed Nov 27 14:08:56 2013] [debug] ssl_engine_kernel.c(1898): OpenSSL: Exit: failed in SSLv3 read client certificate A

Ø  [Wed Nov 27 14:08:56 2013] [info] [client 192.168.124.81] SSL library error 1 in handshake (server 192.168.124.81:443)

Ø  [Wed Nov 27 14:08:56 2013] [info] SSL Library Error: 336151576 error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca

What catches my eye is number 443. I have no idea where that is getting set. Nova is configured on the compute node to respond to  port 8774.

I am also getting an error in the nova/osapi.log file:

[Wed Nov 27 16:50:35 2013] [info] Initial (No.1) HTTPS request received for child 3 (server d00-50-56-8e-79-e7.cloudos.org:8774)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 ERROR nova.api.openstack [req-5183001e-8ca2-4f52-9c56-47ced4cf0570 45c1e6999c0145348d889c5184e4cae5 bf916cad55494d548b4a3a5de78b87a6] Caught error: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack Traceback (most recent call last):
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return req.get_response(self.application)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     application, catch_exc_info=False)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return resp(environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return self.app(env, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return resp(environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return resp(environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     response = self.app(environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return resp(environ, start_response)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 890, in __call__
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     content_type, body, accept)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 969, in _process_stack
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     request, action_args)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 863, in post_process_extensions
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     **action_args)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 537, in detail
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 487, in _extend_servers
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     .get_instances_security_groups_bindings(context))
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/quantum_driver.py", line 252, in get_instances_security_groups_bindings
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     ports = quantum.list_ports().get('ports')
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     **_params)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     res = self.get(path, params=params)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     headers=headers, params=params)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     headers=headers, params=params)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack     message=message)
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack QuantumClientException: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[Wed Nov 27 16:50:35 2013] [error] 2013-11-27 16:50:35.617 31236 TRACE nova.api.openstack
10.1.184.2 - - [27/Nov/2013:16:50:35 -0600] "GET /v2/bf916cad55494d548b4a3a5de78b87a6/servers/detail?project_id=bf916cad55494d548b4a3a5de78b87a6 HTTP/1.1" 500 3120 "-" "python-novaclient"
[Wed Nov 27 16:50:35 2013] [debug] ssl_engine_kernel.c(1884): OpenSSL: Write: SSL negotiation finished successfully
[Wed Nov 27 16:50:35 2013] [info] [client 10.1.184.2] Connection closed to child 3 with standard shutdown (server d00-50-56-8e-79-e7.cloudos.org:8774)

Do you have any suggestions? I have also been unable to find any vhost templates for quantum. I have created my own CA and used it to sign server certificates. To enable using a single certificate for multiple IP addresses for the same server, I have implemented alt_names.

Regards,

Mark

From: Jesse Pretorius [mailto:jesse.pretorius at gmail.com]
Sent: Thursday, November 14, 2013 12:43 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Nova SSL Apache2 Question

On 13 November 2013 23:39, Miller, Mark M (EB SW Cloud - R&D - Corvallis) <mark.m.miller at hp.com<mailto:mark.m.miller at hp.com>> wrote:
I finally found a set of web pages that has a working set of configuration files for the major OpenStack services " http://andymc-stack.co.uk/2013/07/apache2-mod_wsgi-openstack-pt-2-nova-api-os-compute-nova-api-ec2/ " by Andy Mc. I skipped ceilometer and have the rest of the services working except quantum with self-signed certificates on a Grizzly-3 OpenStack instance. Now I am stuck trying to figure out how to get quantum to accept self-signed certificates.

My goal is to harden my Grizzly-3 OpenStack instance using SSL and self-signed certificates. Later I will do the same for Havana bits and use real/valid certificates.

I struggled with getting this all to work correctly for a few weeks, then eventually gave up and opted instead to use an Apache reverse proxy to front-end the native services. I just found that using an Apache/wsgi configuration doesn't completely work. It would certainly help if this configuration was implemented into the Openstack testing regime to help all the services become first-class citizens as a wsgi process behind Apache.

I would suggest that you review the wsgi files and vhost templates in the rcbops chef cookbooks for each service. They include my updates to Andy's original blog items to make things work properly.

I found that while Andy's stuff appears to work, it becomes noticeable that it works in a read-only fashion. I managed to get keystone/nova confirmed to work properly, but glance just would not work - I could never upload any images and if caching/management was turned off in the glance service then downloading images didn't work either.

Good luck - if you do get a fully working config it'd be great to get feedback on the adjustments you had to make to get it working.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131126/6e93f6de/attachment.html>