15 Nov
2022
15 Nov
'22
4:36 p.m.
Hi,
This is the server certificate generated by kolla
# openssl x509 -noout -text -in *backend-cert.pem*
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
36:c4:48:24:e7:88:c4:f0:dd:32:b3:d8:e9:b7:c5:17:5c:4e:85:ff
Signature Algorithm: sha256WithRSAEncryption
*Issuer: CN = KollaTestCA Validity Not Before: Oct 14
13:13:04 2022 GMT Not After : Feb 26 13:13:04 2024 GMT*
Subject: C = US, ST = NC, L = RTP, OU = kolla
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b9:f6:f9:83:e6:8c:de:fb:3e:6f:df:23:b9:46:
53:04:52:7a:45:44:6e:9b:cb:cc:30:ab:df:bc:b2:
....
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
* IP Address:20.3.0.23, IP Address:20.3.0.27, IP
Address:20.3.0.31*
And this is the CA certificate generated by Kolla
# openssl x509 -noout -text -in ca*.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
66:c9:c2:c8:fa:45:e7:48:26:a1:48:63:b6:a9:27:1d:dc:74:4a:c3
Signature Algorithm: sha256WithRSAEncryption
* Issuer: CN = KollaTestCA Validity Not Before:
Oct 14 13:12:59 2022 GMT Not After : Aug 3 13:12:59 2025 GMT
Subject: CN = KollaTestCA*
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:ce:6f:91:5a:bf:81:49:b6:eb:d9:99:60:bc:93:
80:ab:59:bb:20:09:33:b5:b0:75:ba:50:90:87:93:
*# openssl verify -verbose -CAfile ca.pem backend-cert.pembackend-cert.pem:
OK*
>From the keystone container I got this :
*(keystone)[root@controllera /]# curl -v
https://dashint.example.com:5000/v3 <https://dashint.example.com:5000/v3>*
* Trying 20.3.0.1...
* TCP_NODELAY set
* *Connected to dashint.example.com <http://dashint.example.com> (20.3.0.1)
port 5000 (#0)*
* ALPN, offering h2
* ALPN, offering http/1.1
** successfully set certificate verify locations:* CAfile:
/etc/pki/tls/certs/ca-bundle.crt*
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
** subject: C=US; ST=NC; L=RTP; OU=kolla* start date: Oct 14 13:13:03
2022 GMT* expire date: Oct 14 13:13:03 2023 GMT* subjectAltName: host
"dashint.example.com <http://dashint.example.com>" matched cert's
"dashint.example.com <http://dashint.example.com>"*
* issuer: CN=KollaTestCA
* SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /v3 HTTP/1.1
> Host: dashint.example.com:5000
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
*< HTTP/1.1 200 OK*
< date: Sat, 22 Oct 2022 15:39:22 GMT
< server: Apache
< content-length: 262
< vary: X-Auth-Token
< x-openstack-request-id: req-88c293c3-7efb-4a12-ac06-21f90e1fdc10
< content-type: application/json
<
* Connection #0 to host dashint.example.com left intact
{"version": {"id": "v3.14", "status": "stable", "updated":
"2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "
https://dashint.example.com:5000/v3/"}], "media-types": [{"base":
"application/json", "type":
"application/vnd.openstack.identity-v3+json"}]}}curl (
https://dashint.example.com:5000/v3): response: 200, time: 0.012871, size:
262
When deploying with the self certificate it's in this task on the first
controller where the problem is triggered :
*TASK [service-ks-register : keystone | Creating services
module_name=os_keystone_service, module_args={'name': '{{ item.name
<http://item.name> }}', 's$rvice_type': '{{ item.type }}', 'description':
'{{ item.description }}', 'region_name': '{{
service_ks_register_region_name }}', 'au$h': '{{ service_ks_register_auth
}}', 'interface': '{{ service_ks_register_interface }}', 'cacert': '{{
service_ks_cacert }}'}] ****
FAILED - RETRYING: [controllera]: keystone | Creating services (5 retries
left).
FAILED - RETRYING: [controllera]: keystone | Creating services (4 retries
left).
FAILED - RETRYING: [controllera]: keystone | Creating services (3 retries
left).
FAILED - RETRYING: [controllera]: keystone | Creating services (2 retries
left).
FAILED - RETRYING: [controllera]: keystone | Creating services (1 retries
left).failed: [controllera] (item={'name': 'keystone', 'service_type':
'identity'}) => {"action": "os_keystone_service", "ansible_loop_var"
: "item", "attempts": 5, "changed": false, "item": {"description":
"Openstack Identity Service", "endpoints": [{"interface": "admin",
"url": "https://dashint.example.com:35357"}, {"interface": "internal",
"url": "https://dashint.example.com:5000"}, {"interface":
"public", "url": "https://dash.example.com:5000"}], "name": "keystone",
"type": "identity"}, "module_stderr": "Failed to discover
available identity versions when contacting
https://dashint.example.com:35357. Attempting to parse version from
URL.\nTraceback (mo
st recent call last):\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
line 710, in urlopen\n chunk
ed=chunked,\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
line 386, in _make_request\n self._val
idate_conn(conn)\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
line 1040, in _validate_conn\n co
nn.connect()\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/connection.py\", line
426, in connect\n tls_in_tls=tls_in_
tls,\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line
450, in ssl_wrap_socket\n sock, context, tls_
in_tls, server_hostname=server_hostname\n File
\"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line
493, in _ssl_
wrap_socket_impl\n return ssl_context.wrap_socket(sock,
server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\",
line 365, in wrap_socket\n _context=self, _session=session)\n File
\"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n se
lf.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in
do_handshake\n self._sslobj.do_handshake()\n File \"/usr
/lib64/python3.6/ssl.py\", line 648, in do_handshake\n
*self._sslobj.do_handshake()\nssl.SSLError: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed*
(_ssl.c:897)\n\nDuring handling of the above exception, another exception
occurred:\n\nTraceback (most rec
ent call last):\n File
\"/opt/ansible/lib/python3.6/site-packages/requests/adapters.py\", line
450, in send\n timeout=timeout
I don't know what this task is, the container is running, what does
mean *service-ks-register
: keystone* ?
Regards.
Le mar. 15 nov. 2022 à 11:54, Eugen Block <eblock@nde.ag> a écrit :
> Okay, I understand. Did you verify if the self-signed cert contains
> everything you require as I wrote in the previous email? Can you paste
> the openssl command output (and mask everything non-public)?
>
> Zitat von wodel youchi <wodel.youchi@gmail.com>:
>
> > Hi,
> > Thanks again,
> >
> > About your question : so with the previous cert it worked but only
> because
> > you had the verification set to false, correct?
> > The answer is : Not exactly.
> >
> > Let me explain, I deployed using a commercial valid certificate, but I
> > configured kolla_verify_tls_backend to false exactly to avoid the
> problem I
> > am facing now. From what I have understood :
> > kolla_verify_tls_backend=false, means : accept the connection even if the
> > verification fails, but apparently it is not the case.
> > And kolla_copy_ca_into_containers was positioned to yes from the
> beginning.
> >
> > What happened is that my certificate expired, and now I am searching for
> a
> > way to install a self-signed certificate while waiting to get the new
> > certificate.
> >
> > I backported the platform a few days before the expiration of the
> > certificate, then I generated the self-signed certificate and I tried to
> > deploy it but without success.
> >
> > Regards.
> >
> > Le lun. 14 nov. 2022 à 14:21, Eugen Block <eblock@nde.ag> a écrit :
> >
> >> Hi,
> >>
> >> > First I want to correct something, the *kolla_verify_tls_backend* was
> >> > positioned to *false* from the beginning, while doing the first
> >> deployment
> >> > with the commercial certificate.
> >>
> >> so with the previous cert it worked but only because you had the
> >> verification set to false, correct?
> >>
> >> > What do you mean by using openssl? Do you mean to execute the command
> >> > inside a container and try to connect to keystone? If yes what is the
> >> > correct command?
> >>
> >> That's one example, yes. Is apache configured correctly to use the
> >> provided certs? In my manual deployment it looks like this (only the
> >> relevant part):
> >>
> >> control01:~ # cat /etc/apache2/vhosts.d/keystone-public.conf
> >> [...]
> >> SSLEngine On
> >> SSLCertificateFile /etc/ssl/servercerts/control01.fqdn.cert.pem
> >> SSLCACertificateFile
> /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT
> >> SSLCertificateKeyFile /etc/ssl/private/control01.fqdn.key.pem
> >> SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
> >>
> >> # HTTP Strict Transport Security (HSTS) enforces that all
> >> communications
> >> # with a server go over SSL. This mitigates the threat from attacks
> >> such
> >> # as SSL-Strip which replaces links on the wire, stripping away
> >> https prefixes
> >> # and potentially allowing an attacker to view confidential
> >> information on the
> >> # wire
> >> Header add Strict-Transport-Security "max-age=15768000"
> >> [...]
> >>
> >> and then test it with:
> >>
> >> ---snip---
> >> control01:~ # curl -v https://control.fqdn:5000/v3
> >> [...]
> >> * ALPN, offering h2
> >> * ALPN, offering http/1.1
> >> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> >> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> >> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> >> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> >> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> >> * TLSv1.3 (IN), TLS handshake, Finished (20):
> >> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> >> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> >> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> >> * ALPN, server accepted to use http/1.1
> >> * Server certificate:
> >> [...]
> >> * subjectAltName: host "control.fqdn" matched cert's "*.fqdn"
> >> * issuer: *******
> >> * SSL certificate verify ok.
> >> > GET /v3 HTTP/1.1
> >> > Host: control.fqdn:5000
> >> > User-Agent: curl/7.66.0
> >> > Accept: */*
> >> >
> >> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> >> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> >> * old SSL session ID is stale, removing
> >> * Mark bundle as not supporting multiuse
> >> < HTTP/1.1 200 OK
> >> [...]
> >> * Connection #0 to host control.fqdn left intact
> >> {"version": {"id": "v3.14", "status": "stable", "updated":
> >> "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href":
> >> "https://control.fqdn:5000/v3/"}], "media-types": [{"base":
> >> "application/json", "type":
> >> "application/vnd.openstack.identity-v3+json"}]}}
> >> ---snip---
> >>
> >> To check the created certificate you could run something like this:
> >>
> >> openssl x509 -in /etc/ssl/servercerts/control01.fqdn.cert.pem -text
> -noout
> >>
> >> and see if the SANs match your control node(s) IP addresses and FQDNs.
> >>
> >> Zitat von wodel youchi <wodel.youchi@gmail.com>:
> >>
> >> > Hi
> >> >
> >> > Thanks for your help.
> >> >
> >> > First I want to correct something, the *kolla_verify_tls_backend* was
> >> > positioned to *false* from the beginning, while doing the first
> >> deployment
> >> > with the commercial certificate.
> >> >
> >> > And yes I have *kolla_copy_ca_into_containers* positioned to *yes*
> from
> >> the
> >> > beginning. And I can see in the nodes that there is a directory named
> >> > certificates in every module's directory in /etc/kolla
> >> >
> >> > What do you mean by using openssl? Do you mean to execute the command
> >> > inside a container and try to connect to keystone? If yes what is the
> >> > correct command?
> >> >
> >> > It seems like something is missing to tell the client side to ignore
> the
> >> > certificate validity, something like the --insecure parameter in the
> >> > openstack cli.
> >> >
> >> > Regards.
> >> >
> >> > On Fri, Nov 11, 2022, 21:21 Eugen Block <eblock@nde.ag> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I'm not familiar with kolla, but the docs also mention this option:
> >> >>
> >> >> kolla_copy_ca_into_containers: "yes"
> >> >>
> >> >> As I understand it the CA cert is required within the containers so
> >> >> they can trust the self-signed certs. At least that's how I configure
> >> >> it in a manually deployed openstack cloud. Do you have that option
> >> >> enabled? If it is enabled, did you verify it with openssl tools?
> >> >>
> >> >> Regards,
> >> >> Eugen
> >> >>
> >> >> Zitat von wodel youchi <wodel.youchi@gmail.com>:
> >> >>
> >> >> > Some help please.
> >> >> >
> >> >> > On Tue, Nov 8, 2022, 14:44 wodel youchi <wodel.youchi@gmail.com>
> >> wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> To deploy Openstack with a self-signed certificate, the
> documentation
> >> >> says
> >> >> >> to generate the certificates using kolla-ansible certificates, to
> >> >> configure
> >> >> >> the support of TLS in globals.yml and to deploy.
> >> >> >>
> >> >> >> I am facing a problem, my old certificate has expired, I want to
> use
> >> a
> >> >> >> self-signed certificate.
> >> >> >> I backported my servers to an older date, then generated a
> >> self-signed
> >> >> >> certificate using kolla, but the deploy/reconfigure won't work,
> they
> >> >> say :
> >> >> >>
> >> >> >> self._sslobj.do_handshake()\n File
> \"/usr/lib64/python3.6/ssl.py\",
> >> >> line
> >> >> >> 648, in do_handshakeself._sslobj.do_handshake()\nssl.SSLError:
> [SSL:
> >> >> >> CERTIFICATE_VERIFY_FAILED certificate verify failed
> >> >> >>
> >> >> >> PS : in my globals.yml i have : *kolla_verify_tls_backend: "yes"*
> >> >> >>
> >> >> >> Regards.
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >>
>
>
>
>
>