[kolla-ansible][Yoga] Install with self-signed certificate
Eugen Block
eblock at nde.ag
Wed Nov 16 08:15:48 UTC 2022
Hi,
so the curl output looks correct. When you say you backported your
servers, what exactly does that mean? If the servers are set back in
time the commercial certificate CA would still refuse validation,
wouldn't it? Or am I misunderstanding things here? I'm just guessing
here because I'm not familiar with kolla. Is there a debug mode for
the deployment to see which certs exactly it tries to validate? Is
there a way to manually deploy the certs and restart the required
services?
Zitat von wodel youchi <wodel.youchi at gmail.com>:
> Hi,
>
> This is the server certificate generated by kolla
>
> # openssl x509 -noout -text -in *backend-cert.pem*
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> 36:c4:48:24:e7:88:c4:f0:dd:32:b3:d8:e9:b7:c5:17:5c:4e:85:ff
> Signature Algorithm: sha256WithRSAEncryption
>
>
>
> *Issuer: CN = KollaTestCA Validity Not Before: Oct 14
> 13:13:04 2022 GMT Not After : Feb 26 13:13:04 2024 GMT*
> Subject: C = US, ST = NC, L = RTP, OU = kolla
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (2048 bit)
> Modulus:
> 00:b9:f6:f9:83:e6:8c:de:fb:3e:6f:df:23:b9:46:
> 53:04:52:7a:45:44:6e:9b:cb:cc:30:ab:df:bc:b2:
> ....
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Alternative Name:
> * IP Address:20.3.0.23, IP Address:20.3.0.27, IP
> Address:20.3.0.31*
>
> And this is the CA certificate generated by Kolla
> # openssl x509 -noout -text -in ca*.pem
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> 66:c9:c2:c8:fa:45:e7:48:26:a1:48:63:b6:a9:27:1d:dc:74:4a:c3
> Signature Algorithm: sha256WithRSAEncryption
>
>
>
>
> * Issuer: CN = KollaTestCA Validity Not Before:
> Oct 14 13:12:59 2022 GMT Not After : Aug 3 13:12:59 2025 GMT
> Subject: CN = KollaTestCA*
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (4096 bit)
> Modulus:
> 00:ce:6f:91:5a:bf:81:49:b6:eb:d9:99:60:bc:93:
> 80:ab:59:bb:20:09:33:b5:b0:75:ba:50:90:87:93:
>
>
>
> *# openssl verify -verbose -CAfile ca.pem backend-cert.pembackend-cert.pem:
> OK*
>
>
> From the keystone container I got this :
> *(keystone)[root at controllera /]# curl -v
> https://dashint.example.com:5000/v3 <https://dashint.example.com:5000/v3>*
> * Trying 20.3.0.1...
> * TCP_NODELAY set
> * *Connected to dashint.example.com <http://dashint.example.com> (20.3.0.1)
> port 5000 (#0)*
> * ALPN, offering h2
> * ALPN, offering http/1.1
>
> ** successfully set certificate verify locations:* CAfile:
> /etc/pki/tls/certs/ca-bundle.crt*
> CApath: none
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.3 (OUT), TLS handshake, [no content] (0):
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> * ALPN, server did not agree to a protocol
> * Server certificate:
>
>
>
> ** subject: C=US; ST=NC; L=RTP; OU=kolla* start date: Oct 14 13:13:03
> 2022 GMT* expire date: Oct 14 13:13:03 2023 GMT* subjectAltName: host
> "dashint.example.com <http://dashint.example.com>" matched cert's
> "dashint.example.com <http://dashint.example.com>"*
> * issuer: CN=KollaTestCA
> * SSL certificate verify ok.
> * TLSv1.3 (OUT), TLS app data, [no content] (0):
>> GET /v3 HTTP/1.1
>> Host: dashint.example.com:5000
>> User-Agent: curl/7.61.1
>> Accept: */*
>>
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> * TLSv1.3 (IN), TLS handshake, [no content] (0):
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> * TLSv1.3 (IN), TLS app data, [no content] (0):
> *< HTTP/1.1 200 OK*
> < date: Sat, 22 Oct 2022 15:39:22 GMT
> < server: Apache
> < content-length: 262
> < vary: X-Auth-Token
> < x-openstack-request-id: req-88c293c3-7efb-4a12-ac06-21f90e1fdc10
> < content-type: application/json
> <
> * Connection #0 to host dashint.example.com left intact
> {"version": {"id": "v3.14", "status": "stable", "updated":
> "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "
> https://dashint.example.com:5000/v3/"}], "media-types": [{"base":
> "application/json", "type":
> "application/vnd.openstack.identity-v3+json"}]}}curl (
> https://dashint.example.com:5000/v3): response: 200, time: 0.012871, size:
> 262
>
>
> When deploying with the self certificate it's in this task on the first
> controller where the problem is triggered :
>
>
>
> *TASK [service-ks-register : keystone | Creating services
> module_name=os_keystone_service, module_args={'name': '{{ item.name
> <http://item.name> }}', 's$rvice_type': '{{ item.type }}', 'description':
> '{{ item.description }}', 'region_name': '{{
> service_ks_register_region_name }}', 'au$h': '{{ service_ks_register_auth
> }}', 'interface': '{{ service_ks_register_interface }}', 'cacert': '{{
> service_ks_cacert }}'}] ****
> FAILED - RETRYING: [controllera]: keystone | Creating services (5 retries
> left).
> FAILED - RETRYING: [controllera]: keystone | Creating services (4 retries
> left).
> FAILED - RETRYING: [controllera]: keystone | Creating services (3 retries
> left).
> FAILED - RETRYING: [controllera]: keystone | Creating services (2 retries
> left).
> FAILED - RETRYING: [controllera]: keystone | Creating services (1 retries
> left).failed: [controllera] (item={'name': 'keystone', 'service_type':
> 'identity'}) => {"action": "os_keystone_service", "ansible_loop_var"
> : "item", "attempts": 5, "changed": false, "item": {"description":
> "Openstack Identity Service", "endpoints": [{"interface": "admin",
> "url": "https://dashint.example.com:35357"}, {"interface": "internal",
> "url": "https://dashint.example.com:5000"}, {"interface":
> "public", "url": "https://dash.example.com:5000"}], "name": "keystone",
> "type": "identity"}, "module_stderr": "Failed to discover
> available identity versions when contacting
> https://dashint.example.com:35357. Attempting to parse version from
> URL.\nTraceback (mo
> st recent call last):\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
> line 710, in urlopen\n chunk
> ed=chunked,\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
> line 386, in _make_request\n self._val
> idate_conn(conn)\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\",
> line 1040, in _validate_conn\n co
> nn.connect()\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/connection.py\", line
> 426, in connect\n tls_in_tls=tls_in_
> tls,\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line
> 450, in ssl_wrap_socket\n sock, context, tls_
> in_tls, server_hostname=server_hostname\n File
> \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line
> 493, in _ssl_
> wrap_socket_impl\n return ssl_context.wrap_socket(sock,
> server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\",
> line 365, in wrap_socket\n _context=self, _session=session)\n File
> \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n se
> lf.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in
> do_handshake\n self._sslobj.do_handshake()\n File \"/usr
> /lib64/python3.6/ssl.py\", line 648, in do_handshake\n
> *self._sslobj.do_handshake()\nssl.SSLError: [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed*
> (_ssl.c:897)\n\nDuring handling of the above exception, another exception
> occurred:\n\nTraceback (most rec
> ent call last):\n File
> \"/opt/ansible/lib/python3.6/site-packages/requests/adapters.py\", line
> 450, in send\n timeout=timeout
>
>
> I don't know what this task is, the container is running, what does
> mean *service-ks-register
> : keystone* ?
>
> Regards.
>
> Le mar. 15 nov. 2022 à 11:54, Eugen Block <eblock at nde.ag> a écrit :
>
>> Okay, I understand. Did you verify if the self-signed cert contains
>> everything you require as I wrote in the previous email? Can you paste
>> the openssl command output (and mask everything non-public)?
>>
>> Zitat von wodel youchi <wodel.youchi at gmail.com>:
>>
>> > Hi,
>> > Thanks again,
>> >
>> > About your question : so with the previous cert it worked but only
>> because
>> > you had the verification set to false, correct?
>> > The answer is : Not exactly.
>> >
>> > Let me explain, I deployed using a commercial valid certificate, but I
>> > configured kolla_verify_tls_backend to false exactly to avoid the
>> problem I
>> > am facing now. From what I have understood :
>> > kolla_verify_tls_backend=false, means : accept the connection even if the
>> > verification fails, but apparently it is not the case.
>> > And kolla_copy_ca_into_containers was positioned to yes from the
>> beginning.
>> >
>> > What happened is that my certificate expired, and now I am searching for
>> a
>> > way to install a self-signed certificate while waiting to get the new
>> > certificate.
>> >
>> > I backported the platform a few days before the expiration of the
>> > certificate, then I generated the self-signed certificate and I tried to
>> > deploy it but without success.
>> >
>> > Regards.
>> >
>> > Le lun. 14 nov. 2022 à 14:21, Eugen Block <eblock at nde.ag> a écrit :
>> >
>> >> Hi,
>> >>
>> >> > First I want to correct something, the *kolla_verify_tls_backend* was
>> >> > positioned to *false* from the beginning, while doing the first
>> >> deployment
>> >> > with the commercial certificate.
>> >>
>> >> so with the previous cert it worked but only because you had the
>> >> verification set to false, correct?
>> >>
>> >> > What do you mean by using openssl? Do you mean to execute the command
>> >> > inside a container and try to connect to keystone? If yes what is the
>> >> > correct command?
>> >>
>> >> That's one example, yes. Is apache configured correctly to use the
>> >> provided certs? In my manual deployment it looks like this (only the
>> >> relevant part):
>> >>
>> >> control01:~ # cat /etc/apache2/vhosts.d/keystone-public.conf
>> >> [...]
>> >> SSLEngine On
>> >> SSLCertificateFile /etc/ssl/servercerts/control01.fqdn.cert.pem
>> >> SSLCACertificateFile
>> /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT
>> >> SSLCertificateKeyFile /etc/ssl/private/control01.fqdn.key.pem
>> >> SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
>> >>
>> >> # HTTP Strict Transport Security (HSTS) enforces that all
>> >> communications
>> >> # with a server go over SSL. This mitigates the threat from attacks
>> >> such
>> >> # as SSL-Strip which replaces links on the wire, stripping away
>> >> https prefixes
>> >> # and potentially allowing an attacker to view confidential
>> >> information on the
>> >> # wire
>> >> Header add Strict-Transport-Security "max-age=15768000"
>> >> [...]
>> >>
>> >> and then test it with:
>> >>
>> >> ---snip---
>> >> control01:~ # curl -v https://control.fqdn:5000/v3
>> >> [...]
>> >> * ALPN, offering h2
>> >> * ALPN, offering http/1.1
>> >> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>> >> * TLSv1.3 (IN), TLS handshake, Server hello (2):
>> >> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>> >> * TLSv1.3 (IN), TLS handshake, Certificate (11):
>> >> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>> >> * TLSv1.3 (IN), TLS handshake, Finished (20):
>> >> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>> >> * TLSv1.3 (OUT), TLS handshake, Finished (20):
>> >> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>> >> * ALPN, server accepted to use http/1.1
>> >> * Server certificate:
>> >> [...]
>> >> * subjectAltName: host "control.fqdn" matched cert's "*.fqdn"
>> >> * issuer: *******
>> >> * SSL certificate verify ok.
>> >> > GET /v3 HTTP/1.1
>> >> > Host: control.fqdn:5000
>> >> > User-Agent: curl/7.66.0
>> >> > Accept: */*
>> >> >
>> >> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>> >> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>> >> * old SSL session ID is stale, removing
>> >> * Mark bundle as not supporting multiuse
>> >> < HTTP/1.1 200 OK
>> >> [...]
>> >> * Connection #0 to host control.fqdn left intact
>> >> {"version": {"id": "v3.14", "status": "stable", "updated":
>> >> "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href":
>> >> "https://control.fqdn:5000/v3/"}], "media-types": [{"base":
>> >> "application/json", "type":
>> >> "application/vnd.openstack.identity-v3+json"}]}}
>> >> ---snip---
>> >>
>> >> To check the created certificate you could run something like this:
>> >>
>> >> openssl x509 -in /etc/ssl/servercerts/control01.fqdn.cert.pem -text
>> -noout
>> >>
>> >> and see if the SANs match your control node(s) IP addresses and FQDNs.
>> >>
>> >> Zitat von wodel youchi <wodel.youchi at gmail.com>:
>> >>
>> >> > Hi
>> >> >
>> >> > Thanks for your help.
>> >> >
>> >> > First I want to correct something, the *kolla_verify_tls_backend* was
>> >> > positioned to *false* from the beginning, while doing the first
>> >> deployment
>> >> > with the commercial certificate.
>> >> >
>> >> > And yes I have *kolla_copy_ca_into_containers* positioned to *yes*
>> from
>> >> the
>> >> > beginning. And I can see in the nodes that there is a directory named
>> >> > certificates in every module's directory in /etc/kolla
>> >> >
>> >> > What do you mean by using openssl? Do you mean to execute the command
>> >> > inside a container and try to connect to keystone? If yes what is the
>> >> > correct command?
>> >> >
>> >> > It seems like something is missing to tell the client side to ignore
>> the
>> >> > certificate validity, something like the --insecure parameter in the
>> >> > openstack cli.
>> >> >
>> >> > Regards.
>> >> >
>> >> > On Fri, Nov 11, 2022, 21:21 Eugen Block <eblock at nde.ag> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I'm not familiar with kolla, but the docs also mention this option:
>> >> >>
>> >> >> kolla_copy_ca_into_containers: "yes"
>> >> >>
>> >> >> As I understand it the CA cert is required within the containers so
>> >> >> they can trust the self-signed certs. At least that's how I configure
>> >> >> it in a manually deployed openstack cloud. Do you have that option
>> >> >> enabled? If it is enabled, did you verify it with openssl tools?
>> >> >>
>> >> >> Regards,
>> >> >> Eugen
>> >> >>
>> >> >> Zitat von wodel youchi <wodel.youchi at gmail.com>:
>> >> >>
>> >> >> > Some help please.
>> >> >> >
>> >> >> > On Tue, Nov 8, 2022, 14:44 wodel youchi <wodel.youchi at gmail.com>
>> >> wrote:
>> >> >> >
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> To deploy Openstack with a self-signed certificate, the
>> documentation
>> >> >> says
>> >> >> >> to generate the certificates using kolla-ansible certificates, to
>> >> >> configure
>> >> >> >> the support of TLS in globals.yml and to deploy.
>> >> >> >>
>> >> >> >> I am facing a problem, my old certificate has expired, I want to
>> use
>> >> a
>> >> >> >> self-signed certificate.
>> >> >> >> I backported my servers to an older date, then generated a
>> >> self-signed
>> >> >> >> certificate using kolla, but the deploy/reconfigure won't work,
>> they
>> >> >> say :
>> >> >> >>
>> >> >> >> self._sslobj.do_handshake()\n File
>> \"/usr/lib64/python3.6/ssl.py\",
>> >> >> line
>> >> >> >> 648, in do_handshakeself._sslobj.do_handshake()\nssl.SSLError:
>> [SSL:
>> >> >> >> CERTIFICATE_VERIFY_FAILED certificate verify failed
>> >> >> >>
>> >> >> >> PS : in my globals.yml i have : *kolla_verify_tls_backend: "yes"*
>> >> >> >>
>> >> >> >> Regards.
>> >> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >>
>> >>
>>
>>
>>
>>
>>
More information about the openstack-discuss
mailing list