<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Quick follow-up, just to close the loop on this thread. I found that there had been a lot of recent code changes/improvements in Cloudkitty 8.x, which is from the Rocky release. So on a hunch, I decided to see if I could run the Rocky version of Cloudkitty with the rest of OpenStack Queens. On CentOS 7.5, you need to upgrade python2-six from version 1.10 to 1.11 -- that's the only RPM dependency thing, and the change seems to have no effect on the rest of Queens. Other than that, Cloudkitty from Rocky installs just fine on CentOS 7.5 controller nodes running other parts of Queens. The packages (from <a href="http://mirror.centos.org/centos/7/cloud/x86_64/openstack-rocky">http://mirror.centos.org/centos/7/cloud/x86_64/openstack-rocky</a>) are:</div><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr">openstack-cloudkitty-api-8.0.0-1.el7.noarch</div><div dir="ltr">openstack-cloudkitty-common-8.0.0-1.el7.noarch</div><div dir="ltr">openstack-cloudkitty-processor-8.0.0-1.el7.noarch</div><div dir="ltr">openstack-cloudkitty-ui-8.0.0-1.el7.noarch</div><div dir="ltr">openstack-cloudkitty-ui-doc-8.0.0-1.el7.noarch</div><div dir="ltr">python2-cloudkittyclient-2.0.0-1.el7.noarch</div><div><br></div><div>After the upgrade, I saw immediate improvement in Cloudkitty's handling of SSL, but there was still a snag with the Horizon dashboard plugin. The folks at Objectif Libre (Christophe, Luka, Sebastien) have been working with me on the problem: I sent a bunch of debug output yesterday, and this afternoon they produced the following patches:</div><div><br></div><div><a href="https://review.openstack.org/#/c/600510/">https://review.openstack.org/#/c/600510/</a><br></div><div><a href="https://review.openstack.org/#/c/600515/">https://review.openstack.org/#/c/600515/</a><br></div><div><br></div><div>According to my testing, everything now works! That was a fast turnaround! So my thanks, again, to Christophe and Objectif Libre. If you are going to be at SC18, feel free to stop by the NASA booth and you may well get to see their software in action.</div><div><br></div><div>Jonathan Mills</div><div>NASA Goddard Space Flight Center</div><div><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Sep 4, 2018 at 8:37 AM Jonathan Mills <<a href="mailto:jonmills@gmail.com">jonmills@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Christophe,<div><br></div><div>Thank you, we really appreciate you looking into this, and I will try to help you as much as I can, because we really need to have this software working, soon.</div><div><br></div><div>So here's something that, to me, is very telling</div><div><br></div><div><div># printenv |grep OS_CACERT</div><div>OS_CACERT=/etc/openldap/cacerts/gpcprod_root_ca.pem</div></div><div><br></div><div>^^^ here you can see that my self-signed CA cert is loaded into my environment, having sourced my openrc file</div><div><br></div><div>Now I'm going to invoke the cloudkitty client with debug, and grep for 'curl' to see what it's actually doing:</div><div><br></div><div><div># openstack --debug rating info-config-get 2>&1 |grep -b1 curl</div><div>9774-Get auth_ref</div><div>9787:REQ: curl -g -i --cacert "/etc/openldap/cacerts/gpcprod_root_ca.pem" -X GET <a href="https://keystone.gpcprod:5000/v3" target="_blank">https://keystone.gpcprod:5000/v3</a> -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"</div><div>10014-Starting new HTTPS connection (1): keystone.gpcprod</div><div>--</div><div>16319-run(Namespace())</div><div>16336:REQ: curl -g -i -X GET <a href="https://keystone.gpcprod:5000/v3" target="_blank">https://keystone.gpcprod:5000/v3</a> -H "Accept: application/json" -H "User-Agent: python-keystoneclient"</div><div>16461-Starting new HTTPS connection (1): keystone.gpcprod</div></div><div><br></div><div>^^^ you can see that the first time, it correctly forms the curl, and that works fine. But the second time (and the User-Agent has changed), it never even passes the --cacert option to curl at all. The results then are predictable:</div><div><br></div><div><div>Starting new HTTPS connection (1): keystone.gpcprod</div><div>SSL exception connecting to <a href="https://keystone.gpcprod:5000/v3" target="_blank">https://keystone.gpcprod:5000/v3</a>: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand</div><div> result = cmd.run(parsed_args)</div><div> File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run</div><div> return super(Command, self).run(parsed_args)</div><div> File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run</div><div> return_code = self.take_action(parsed_args) or 0</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell_cli.py", line 78, in take_action</div><div> shell.do_info_config_get(ckclient, parsed_args)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell.py", line 93, in do_info_config_get</div><div> utils.print_dict(cc.config.get_config(), dict_property="Section")</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/core.py", line 88, in get_config</div><div> out = self.api.get(self.base_url).json()</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py", line 359, in get</div><div> return self.client_request("GET", url, **kwargs)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py", line 349, in client_request</div><div> self, method, url, **kwargs)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py", line 248, in client_request</div><div> self.authenticate()</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py", line 319, in authenticate</div><div> self.auth_plugin.authenticate(self)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/auth.py", line 201, in authenticate</div><div> self._do_authenticate(http_client)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line 191, in _do_authenticate</div><div> ks_session = _get_keystone_session(**ks_kwargs)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line 87, in _get_keystone_session</div><div> v2_auth_url, v3_auth_url = _discover_auth_versions(ks_session, auth_url)</div><div> File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line 38, in _discover_auth_versions</div><div> ks_discover = discover.Discover(session=session, auth_url=auth_url)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/discover.py", line 178, in __init__</div><div> authenticated=authenticated)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line 143, in __init__</div><div> authenticated=authenticated)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line 38, in get_version_data</div><div> resp = session.get(url, headers=headers, authenticated=authenticated)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 535, in get</div><div> return self.request(url, 'GET', **kwargs)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 428, in request</div><div> resp = send(**kwargs)</div><div> File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 466, in _send_request</div><div> raise exceptions.SSLError(msg)</div><div>SSLError: SSL exception connecting to <a href="https://keystone.gpcprod:5000/v3" target="_blank">https://keystone.gpcprod:5000/v3</a>: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)</div><div>clean_up CliInfoGetConfig: SSL exception connecting to <a href="https://keystone.gpcprod:5000/v3" target="_blank">https://keystone.gpcprod:5000/v3</a>: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)</div></div><div><br></div><div><br></div><div>Jonathan</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Sep 4, 2018 at 5:50 AM Christophe Sauthier <<a href="mailto:christophe.sauthier@objectif-libre.com" target="_blank">christophe.sauthier@objectif-libre.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello<br>
<br>
Thanks for those elements.<br>
<br>
It is really surprising because as you can imagine this is something we <br>
set up many times...<br>
I'll take care to set up the same environment than you and I'll let you <br>
know if I am facing the same issues... I am trying to do that quickly...<br>
<br>
Regards<br>
<br>
Christophe<br>
<br>
----<br>
Christophe Sauthier<br>
CEO<br>
<br>
Objectif Libre : Au service de votre Cloud<br>
<br>
+33 (0) 6 16 98 63 96 | <a href="mailto:christophe.sauthier@objectif-libre.com" target="_blank">christophe.sauthier@objectif-libre.com</a><br>
<br>
<a href="https://www.objectif-libre.com" rel="noreferrer" target="_blank">https://www.objectif-libre.com</a> | @objectiflibre<br>
Recevez la Pause Cloud Et DevOps : <a href="https://olib.re/abo-pause" rel="noreferrer" target="_blank">https://olib.re/abo-pause</a><br>
<br>
Le 2018-08-31 23:40, <a href="mailto:jonmills@gmail.com" target="_blank">jonmills@gmail.com</a> a écrit :<br>
> On Fri, 2018-08-31 at 23:20 +0200, Christophe Sauthier wrote:<br>
>> Hello Jonathan<br>
>> <br>
>> Can you describe a little more your setup (release/method of<br>
>> installation/linux distribution) /issues that you are facing ?<br>
> <br>
> <br>
> It is OpenStack Queens, on CentOS 7.5, using the packages from the<br>
> centos-cloud repo (which I suppose is the same is RDO).<br>
> <br>
> # uname -msr<br>
> Linux 3.10.0-862.3.2.el7.x86_64 x86_64<br>
> <br>
> # rpm -qa |grep cloudkitty |sort<br>
> openstack-cloudkitty-api-7.0.0-1.el7.noarch<br>
> openstack-cloudkitty-common-7.0.0-1.el7.noarch<br>
> openstack-cloudkitty-processor-7.0.0-1.el7.noarch<br>
> openstack-cloudkitty-ui-7.0.0-1.el7.noarch<br>
> python2-cloudkittyclient-1.2.0-1.el7.noarch<br>
> <br>
> It is 'deployed' with custom puppet code only. I follow exactly the<br>
> installation guides posted here:<br>
> <a href="https://docs.openstack.org/cloudkitty/queens/index.html" rel="noreferrer" target="_blank">https://docs.openstack.org/cloudkitty/queens/index.html</a><br>
> <br>
> I'd prefer not to post full config files, but my [keystone_authtoken]<br>
> section of cloudkitty.conf is identical (aside from service<br>
> credentials) to the ones found in my glance, nova, cinder, neutron,<br>
> gnocchi, ceilometer, etc, all of those services are working perfectly.<br>
> <br>
> <br>
> My processor.log file is full of<br>
> <br>
> 2018-08-31 16:38:04.086 30471 WARNING cloudkitty.orchestrator [-] <br>
> Error<br>
> while collecting service network.floating: SSL exception connecting to<br>
> <a href="https://keystone.gpcprod:5000/v3/auth/tokens" rel="noreferrer" target="_blank">https://keystone.gpcprod:5000/v3/auth/tokens</a>: ("bad handshake:<br>
> Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate<br>
> verify failed')],)",): SSLError: SSL exception connecting to<br>
> <a href="https://keystone.gpcprod:5000/v3/auth/tokens" rel="noreferrer" target="_blank">https://keystone.gpcprod:5000/v3/auth/tokens</a>: ("bad handshake:<br>
> Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate<br>
> verify failed')],)",)<br>
> 2018-08-31 16:38:04.094 30471 WARNING cloudkitty.orchestrator [-] <br>
> Error<br>
> while collecting service image: SSL exception connecting to<br>
> <a href="https://keystone.gpcprod:5000/v3/auth/tokens" rel="noreferrer" target="_blank">https://keystone.gpcprod:5000/v3/auth/tokens</a>: ("bad handshake:<br>
> Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate<br>
> verify failed')],)",): SSLError: SSL exception connecting to<br>
> <a href="https://keystone.gpcprod:5000/v3/auth/tokens" rel="noreferrer" target="_blank">https://keystone.gpcprod:5000/v3/auth/tokens</a>: ("bad handshake:<br>
> Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate<br>
> verify failed')],)",)<br>
> <br>
> and so on<br>
> <br>
> <br>
> But, I mean, there's other little things too. I can see from running<br>
> <br>
> 'openstack --debug rating info-config-get'<br>
> <br>
> that it never even loads the cacert from my env, so it fails talking <br>
> to<br>
> keystone trying to get a token; the request never even gets to the<br>
> cloudkitty api endpoint.<br>
> <br>
> <br>
> <br>
>> <br>
>> Because we have deployed it/used it many times with SSL without<br>
>> issue...<br>
>> <br>
>> It could be great also that you step up on #cloudkitty to discuss it.<br>
>> <br>
>> Christophe<br>
>> <br>
>> ----<br>
>> Christophe Sauthier<br>
>> CEO<br>
>> <br>
>> Objectif Libre : Au service de votre Cloud<br>
>> <br>
>> +33 (0) 6 16 98 63 96 | <a href="mailto:christophe.sauthier@objectif-libre.com" target="_blank">christophe.sauthier@objectif-libre.com</a><br>
>> <br>
>> <a href="https://www.objectif-libre.com" rel="noreferrer" target="_blank">https://www.objectif-libre.com</a> | @objectiflibre<br>
>> Recevez la Pause Cloud Et DevOps : <a href="https://olib.re/abo-pause" rel="noreferrer" target="_blank">https://olib.re/abo-pause</a><br>
>> <br>
>> Le 2018-08-31 23:15, <a href="mailto:jonmills@gmail.com" target="_blank">jonmills@gmail.com</a> a écrit :<br>
>>> Anyone out there have Cloudkitty successfully working with SSL? By<br>
>>> which I mean that Cloudkitty is able to talk to keystone over https<br>
>>> without cert errors, and also talk to SSL'd rabbitmq? Oh, and the<br>
>>> client tools also?<br>
>>> <br>
>>> Asking for a friend...<br>
>>> <br>
>>> <br>
>>> <br>
>>> Jonathan<br>
>>> <br>
>>> <br>
>>> _______________________________________________<br>
>>> OpenStack-operators mailing list<br>
>>> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
>>> <br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote></div>
</blockquote></div>