Open Stack

Christophe,

Thank you, we really appreciate you looking into this, and I will try to
help you as much as I can, because we really need to have this software
working, soon.

So here's something that, to me, is very telling

# printenv |grep OS_CACERT
OS_CACERT=/etc/openldap/cacerts/gpcprod_root_ca.pem

^^^ here you can see that my self-signed CA cert is loaded into my
environment, having sourced my openrc file

Now I'm going to invoke the cloudkitty client with debug, and grep for
'curl' to see what it's actually doing:

# openstack --debug rating info-config-get 2>&1 |grep -b1 curl
9774-Get auth_ref
9787:REQ: curl -g -i --cacert "/etc/openldap/cacerts/gpcprod_root_ca.pem"
-X GET https://keystone.gpcprod:5000/v3 -H "Accept: application/json" -H
"User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2
CPython/2.7.5"
10014-Starting new HTTPS connection (1): keystone.gpcprod
--
16319-run(Namespace())
16336:REQ: curl -g -i -X GET https://keystone.gpcprod:5000/v3 -H "Accept:
application/json" -H "User-Agent: python-keystoneclient"
16461-Starting new HTTPS connection (1): keystone.gpcprod

^^^ you can see that the first time, it correctly forms the curl, and that
works fine.  But the second time (and the User-Agent has changed), it never
even passes the --cacert option to curl at all.  The results then are
predictable:

Starting new HTTPS connection (1): keystone.gpcprod
SSL exception connecting to https://keystone.gpcprod:5000/v3: ("bad
handshake: Error([('SSL routines', 'ssl3_get_server_certificate',
'certificate verify failed')],)",)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in
run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line
41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell_cli.py",
line 78, in take_action
    shell.do_info_config_get(ckclient, parsed_args)
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/shell.py",
line 93, in do_info_config_get
    utils.print_dict(cc.config.get_config(), dict_property="Section")
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/v1/core.py", line
88, in get_config
    out = self.api.get(self.base_url).json()
  File
"/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
line 359, in get
    return self.client_request("GET", url, **kwargs)
  File
"/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
line 349, in client_request
    self, method, url, **kwargs)
  File
"/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
line 248, in client_request
    self.authenticate()
  File
"/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/client.py",
line 319, in authenticate
    self.auth_plugin.authenticate(self)
  File
"/usr/lib/python2.7/site-packages/cloudkittyclient/apiclient/auth.py", line
201, in authenticate
    self._do_authenticate(http_client)
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
191, in _do_authenticate
    ks_session = _get_keystone_session(**ks_kwargs)
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
87, in _get_keystone_session
    v2_auth_url, v3_auth_url = _discover_auth_versions(ks_session, auth_url)
  File "/usr/lib/python2.7/site-packages/cloudkittyclient/client.py", line
38, in _discover_auth_versions
    ks_discover = discover.Discover(session=session, auth_url=auth_url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/discover.py", line
178, in __init__
    authenticated=authenticated)
  File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line
143, in __init__
    authenticated=authenticated)
  File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line
38, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
535, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
428, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line
466, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://keystone.gpcprod:5000/v3:
("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate',
'certificate verify failed')],)",)
clean_up CliInfoGetConfig: SSL exception connecting to
https://keystone.gpcprod:5000/v3: ("bad handshake: Error([('SSL routines',
'ssl3_get_server_certificate', 'certificate verify failed')],)",)


Jonathan

On Tue, Sep 4, 2018 at 5:50 AM Christophe Sauthier <
christophe.sauthier at objectif-libre.com> wrote:

> Hello
>
> Thanks for those elements.
>
> It is really surprising because as you can imagine this is something we
> set up many times...
> I'll take care to set up the same environment than you and I'll let you
> know if I am facing the same issues... I am trying to do that quickly...
>
> Regards
>
>      Christophe
>
> ----
> Christophe Sauthier
> CEO
>
> Objectif Libre : Au service de votre Cloud
>
> +33 (0) 6 16 98 63 96 | christophe.sauthier at objectif-libre.com
>
> https://www.objectif-libre.com | @objectiflibre
> Recevez la Pause Cloud Et DevOps : https://olib.re/abo-pause
>
> Le 2018-08-31 23:40, jonmills at gmail.com a écrit :
> > On Fri, 2018-08-31 at 23:20 +0200, Christophe Sauthier wrote:
> >> Hello Jonathan
> >>
> >> Can you describe a little more your setup (release/method of
> >> installation/linux distribution) /issues that you are facing ?
> >
> >
> > It is OpenStack Queens, on CentOS 7.5, using the packages from the
> > centos-cloud repo (which I suppose is the same is RDO).
> >
> > # uname -msr
> > Linux 3.10.0-862.3.2.el7.x86_64 x86_64
> >
> > # rpm -qa |grep cloudkitty |sort
> > openstack-cloudkitty-api-7.0.0-1.el7.noarch
> > openstack-cloudkitty-common-7.0.0-1.el7.noarch
> > openstack-cloudkitty-processor-7.0.0-1.el7.noarch
> > openstack-cloudkitty-ui-7.0.0-1.el7.noarch
> > python2-cloudkittyclient-1.2.0-1.el7.noarch
> >
> > It is 'deployed' with custom puppet code only.  I follow exactly the
> > installation guides posted here:
> > https://docs.openstack.org/cloudkitty/queens/index.html
> >
> > I'd prefer not to post full config files, but my [keystone_authtoken]
> > section of cloudkitty.conf is identical (aside from service
> > credentials) to the ones found in my glance, nova, cinder, neutron,
> > gnocchi, ceilometer, etc, all of those services are working perfectly.
> >
> >
> > My processor.log file is full of
> >
> > 2018-08-31 16:38:04.086 30471 WARNING cloudkitty.orchestrator [-]
> > Error
> > while collecting service network.floating: SSL exception connecting to
> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
> > verify failed')],)",): SSLError: SSL exception connecting to
> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
> > verify failed')],)",)
> > 2018-08-31 16:38:04.094 30471 WARNING cloudkitty.orchestrator [-]
> > Error
> > while collecting service image: SSL exception connecting to
> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
> > verify failed')],)",): SSLError: SSL exception connecting to
> > https://keystone.gpcprod:5000/v3/auth/tokens: ("bad handshake:
> > Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate
> > verify failed')],)",)
> >
> > and so on
> >
> >
> > But, I mean, there's other little things too.  I can see from running
> >
> > 'openstack --debug rating info-config-get'
> >
> > that it never even loads the cacert from my env, so it fails talking
> > to
> > keystone trying to get a token; the request never even gets to the
> > cloudkitty api endpoint.
> >
> >
> >
> >>
> >> Because we have deployed it/used it many times with SSL without
> >> issue...
> >>
> >> It could be great also that you step up on #cloudkitty to discuss it.
> >>
> >>       Christophe
> >>
> >> ----
> >> Christophe Sauthier
> >> CEO
> >>
> >> Objectif Libre : Au service de votre Cloud
> >>
> >> +33 (0) 6 16 98 63 96 | christophe.sauthier at objectif-libre.com
> >>
> >> https://www.objectif-libre.com | @objectiflibre
> >> Recevez la Pause Cloud Et DevOps : https://olib.re/abo-pause
> >>
> >> Le 2018-08-31 23:15, jonmills at gmail.com a écrit :
> >>> Anyone out there have Cloudkitty successfully working with SSL?  By
> >>> which I mean that Cloudkitty is able to talk to keystone over https
> >>> without cert errors, and also talk to SSL'd rabbitmq?  Oh, and the
> >>> client tools also?
> >>>
> >>> Asking for a friend...
> >>>
> >>>
> >>>
> >>> Jonathan
> >>>
> >>>
> >>> _______________________________________________
> >>> OpenStack-operators mailing list
> >>> OpenStack-operators at lists.openstack.org
> >>>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180904/e0808974/attachment.html>