Open Stack

Quick follow-up, just to close the loop on this thread.  I found that there
had been a lot of recent code changes/improvements in Cloudkitty 8.x, which
is from the Rocky release.  So on a hunch, I decided to see if I could run
the Rocky version of Cloudkitty with the rest of OpenStack Queens.  On
CentOS 7.5, you need to upgrade python2-six from version 1.10 to 1.11 --
that's the only RPM dependency thing, and the change seems to have no
effect on the rest of Queens.  Other than that, Cloudkitty from Rocky
installs just fine on CentOS 7.5 controller nodes running other parts of
Queens.  The packages (from
http://mirror.centos.org/centos/7/cloud/x86_64/openstack-rocky) are:

openstack-cloudkitty-api-8.0.0-1.el7.noarch
openstack-cloudkitty-common-8.0.0-1.el7.noarch
openstack-cloudkitty-processor-8.0.0-1.el7.noarch
openstack-cloudkitty-ui-8.0.0-1.el7.noarch
openstack-cloudkitty-ui-doc-8.0.0-1.el7.noarch
python2-cloudkittyclient-2.0.0-1.el7.noarch

After the upgrade, I saw immediate improvement in Cloudkitty's handling of
SSL, but there was still a snag with the Horizon dashboard plugin.  The
folks at Objectif Libre (Christophe, Luka, Sebastien) have been working
with me on the problem: I sent a bunch of debug output yesterday, and this
afternoon they produced the following patches:

https://review.openstack.org/#/c/600510/
https://review.openstack.org/#/c/600515/

According to my testing, everything now works!  That was a fast
turnaround!  So my thanks, again, to Christophe and Objectif Libre.  If you
are going to be at SC18, feel free to stop by the NASA booth and you may
well get to see their software in action.

Jonathan Mills
NASA Goddard Space Flight Center


On Tue, Sep 4, 2018 at 8:37 AM Jonathan Mills <jonmills at gmail.com> wrote:

> Christophe,
>
> Thank you, we really appreciate you looking into this, and I will try to
> help you as much as I can, because we really need to have this software
> working, soon.
>
> So here's something that, to me, is very telling
>
> # printenv |grep OS_CACERT
> OS_CACERT=/etc/openldap/cacerts/gpcprod_root_ca.pem
>
> ^^^ here you can see that my self-signed CA cert is loaded into my
> environment, having sourced my openrc file
>
> Now I'm going to invoke the cloudkitty client with debug, and grep for
> 'curl' to see what it's actually doing:
>
> # openstack --debug rating info-config-get 2>&1 |grep -b1 curl
> 9774-Get auth_ref
> 9787:REQ: curl -g -i --cacert "/etc/openldap/cacerts/gpcprod_root_ca.pem"
> -X GET https://keystone.gpcprod:5000/v3 -H "Accept: application/json" -H
> "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2
> CPython/2.7.5"
> 10014-Starting new HTTPS connection (1): keystone.gpcprod
> --
> 16319-run(Namespace())
> 16336:REQ: curl -g -i -X GET https://keystone.gpcprod:5000/v3 -H "Accept:
> application/json" -H "User-Agent: python-keystoneclient"
> 16461-Starting new HTTPS connection (1): keystone.gpcprod
>
> ^^^ you can see that the first time, it correctly forms the curl, and that
> works fine.  But the second time (and the User-Agent has changed), it never
> even passes the --cacert option to curl at all.  The results then are
> predictable:
>
> Starting new HTTPS connection (1): keystone.gpcprod
> SSL exception connecting to https://keystone.gpcprod:5000/v3: ("bad
> handshake: Error([('SSL routines', 'ssl3_get_server_certificate',
> 'certificate verify failed')],)",)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in
> run_subcommand
>     result = cmd.run(parsed_args)
>   File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line
> 41, in run
>     return super(Command, self).run(parsed_args)
>   File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in
> run
>     return_code = self.take_action(parsed_args) or 0
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell_cli.py", line
> 78, in take_action
>     shell.do_info_config_get(ckclient, parsed_args)
>   File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell.py",
> line 93, in do_info_config_get
>     utils.print_dict(cc.config.get_config(), dict_property="Section")
>   File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/core.py",
> line 88, in get_config
>     out = self.api.get(self.base_url).json()
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
> line 359, in get
>     return self.client_request("GET", url, **kwargs)
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
> line 349, in client_request
>     self, method, url, **kwargs)
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
> line 248, in client_request
>     self.authenticate()
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
> line 319, in authenticate
>     self.auth_plugin.authenticate(self)
>   File
> "/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/auth.py", line
> 201, in authenticate
>     self._do_authenticate(http_client)
>   File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
> 191, in _do_authenticate
>     ks_session = _get_keystone_session(**ks_kwargs)
>   File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
> 87, in _get_keystone_session
>     v2_auth_url, v3_auth_url = _discover_auth_versions(ks_session,
> auth_url)
>   File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
> 38, in _discover_auth_versions
>     ks_discover = discover.Discover(session=session, auth_url=auth_url)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/discover.py", line
> 178, in __init__
>     authenticated=authenticated)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py",
> line 143, in __init__
>     authenticated=authenticated)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py",
> line 38, in get_version_data
>     resp = session.get(url, headers=headers, authenticated=authenticated)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
> 535, in get
>     return self.request(url, 'GET', **kwargs)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
> 428, in request
>     resp = send(**kwargs)
>   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
> 466, in _send_request
>     raise exceptions.SSLError(msg)
> SSLError: SSL exception connecting to https://keystone.gpcprod:5000/v3:
> ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate',
> 'certificate verify failed')],)",)
> clean_up CliInfoGetConfig: SSL exception connecting to
> https://keystone.gpcprod:5000/v3: ("bad handshake: Error([('SSL
> routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
>
>
> Jonathan
>
> On Tue, Sep 4, 2018 at 5:50 AM Christophe Sauthier <
> christophe.sauthier at objectif-libre.com> wrote:
>
>> Hello
>>
>> Thanks for those elements.
>>
>> It is really surprising because as you can imagine this is something we
>> set up many times...
>> I'll take care to set up the same environment than you and I'll let you
>> know if I am facing the same issues... I am trying to do that quickly...
>>
>> Regards
>>
>>      Christophe
>>
>> ----
>> Christophe Sauthier
>> CEO
>>
>> Objectif Libre : Au service de votre Cloud
>>
>> +33 (0) 6 16 98 63 96 | christophe.sauthier at objectif-libre.com
>>
>> https://www.objectif-libre.com | @objectiflibre
>> Recevez la Pause Cloud Et DevOps : https://olib.re/abo-pause
>>
>> Le 2018-08-31 23:40, jonmills at gmail.com a écrit :
>> > On Fri, 2018-08-31 at 23:20 +0200, Christophe Sauthier wrote:
>> >> Hello Jonathan
>> >>
>> >> Can you describe a little more your setup (release/method of
>> >> installation/linux distribution) /issues that you are facing ?
>> >
>> >
>> > It is OpenStack Queens, on CentOS 7.5, using the packages from the
>> > centos-cloud repo (which I suppose is the same is RDO).
>> >
>> > # uname -msr
>> > Linux 3.10.0-862.3.2.el7.x86_64 x86_64
>> >
>> > # rpm -qa |grep cloudkitty |sort
>> > openstack-cloudkitty-api-7.0.0-1.el7.noarch
>> > openstack-cloudkitty-common-7.0.0-1.el7.noarch
>> > openstack-cloudkitty-processor-7.0.0-1.el7.noarch
>> > openstack-cloudkitty-ui-7.0.0-1.el7.noarch
>> > python2-cloudkittyclient-1.2.0-1.el7.noarch
>> >
>> > It is 'deployed' with custom puppet code only.  I follow exactly the
>> > installation guides posted here:
>> > https://docs.openstack.org/cloudkitty/queens/index.html
>> >
>> > I'd prefer not to post full config files, but my [keystone_authtoken]
>> > section of cloudkitty.conf is identical (aside from service
>> > credentials) to the ones found in my glance, nova, cinder, neutron,
>> > gnocchi, ceilometer, etc, all of those services are working perfectly.
>> >
>> >
>> > My processor.log file is full of
>> >
>> > 2018-08-31 16:38:04.086 30471 WARNING cloudkitty.orchestrator [-]
>> > Error
>> > while collecting service network.floating: SSL exception connecting to
>> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
>> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
>> > verify failed')],)",): SSLError: SSL exception connecting to
>> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
>> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
>> > verify failed')],)",)
>> > 2018-08-31 16:38:04.094 30471 WARNING cloudkitty.orchestrator [-]
>> > Error
>> > while collecting service image: SSL exception connecting to
>> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
>> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
>> > verify failed')],)",): SSLError: SSL exception connecting to
>> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
>> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
>> > verify failed')],)",)
>> >
>> > and so on
>> >
>> >
>> > But, I mean, there's other little things too.  I can see from running
>> >
>> > 'openstack --debug rating info-config-get'
>> >
>> > that it never even loads the cacert from my env, so it fails talking
>> > to
>> > keystone trying to get a token; the request never even gets to the
>> > cloudkitty api endpoint.
>> >
>> >
>> >
>> >>
>> >> Because we have deployed it/used it many times with SSL without
>> >> issue...
>> >>
>> >> It could be great also that you step up on #cloudkitty to discuss it.
>> >>
>> >>       Christophe
>> >>
>> >> ----
>> >> Christophe Sauthier
>> >> CEO
>> >>
>> >> Objectif Libre : Au service de votre Cloud
>> >>
>> >> +33 (0) 6 16 98 63 96 | christophe.sauthier at objectif-libre.com
>> >>
>> >> https://www.objectif-libre.com | @objectiflibre
>> >> Recevez la Pause Cloud Et DevOps : https://olib.re/abo-pause
>> >>
>> >> Le 2018-08-31 23:15, jonmills at gmail.com a écrit :
>> >>> Anyone out there have Cloudkitty successfully working with SSL?  By
>> >>> which I mean that Cloudkitty is able to talk to keystone over https
>> >>> without cert errors, and also talk to SSL'd rabbitmq?  Oh, and the
>> >>> client tools also?
>> >>>
>> >>> Asking for a friend...
>> >>>
>> >>>
>> >>>
>> >>> Jonathan
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> OpenStack-operators mailing list
>> >>> OpenStack-operators at lists.openstack.org
>> >>>
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180906/bde14a6e/attachment.html>