Hi,

This is the server certificate generated by kolla

 # openssl x509 -noout -text -in backend-cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            36:c4:48:24:e7:88:c4:f0:dd:32:b3:d8:e9:b7:c5:17:5c:4e:85:ff
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = KollaTestCA
        Validity
            Not Before: Oct 14 13:13:04 2022 GMT
            Not After : Feb 26 13:13:04 2024 GMT

        Subject: C = US, ST = NC, L = RTP, OU = kolla
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b9:f6:f9:83:e6:8c:de:fb:3e:6f:df:23:b9:46:
                    53:04:52:7a:45:44:6e:9b:cb:cc:30:ab:df:bc:b2:
                    ....
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                IP Address:20.3.0.23, IP Address:20.3.0.27, IP Address:20.3.0.31

And this is the CA certificate generated by Kolla
# openssl x509 -noout -text -in ca*.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            66:c9:c2:c8:fa:45:e7:48:26:a1:48:63:b6:a9:27:1d:dc:74:4a:c3
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = KollaTestCA
        Validity
            Not Before: Oct 14 13:12:59 2022 GMT
            Not After : Aug  3 13:12:59 2025 GMT
        Subject: CN = KollaTestCA

        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:ce:6f:91:5a:bf:81:49:b6:eb:d9:99:60:bc:93:
                    80:ab:59:bb:20:09:33:b5:b0:75:ba:50:90:87:93:


# openssl verify -verbose -CAfile ca.pem backend-cert.pem
backend-cert.pem: OK


From the keystone container I got this :
(keystone)[root@controllera /]# curl -v https://dashint.example.com:5000/v3
*   Trying 20.3.0.1...
* TCP_NODELAY set
* Connected to dashint.example.com (20.3.0.1) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=NC; L=RTP; OU=kolla
*  start date: Oct 14 13:13:03 2022 GMT
*  expire date: Oct 14 13:13:03 2023 GMT
*  subjectAltName: host "dashint.example.com" matched cert's "dashint.example.com"

*  issuer: CN=KollaTestCA
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /v3 HTTP/1.1
> Host: dashint.example.com:5000
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< date: Sat, 22 Oct 2022 15:39:22 GMT
< server: Apache
< content-length: 262
< vary: X-Auth-Token
< x-openstack-request-id: req-88c293c3-7efb-4a12-ac06-21f90e1fdc10
< content-type: application/json
<
* Connection #0 to host dashint.example.com left intact
{"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://dashint.example.com:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}curl (https://dashint.example.com:5000/v3): response: 200, time: 0.012871, size: 262


When deploying with the self certificate it's in this task on the first controller where the problem is triggered :

TASK [service-ks-register : keystone | Creating services module_name=os_keystone_service, module_args={'name': '{{ item.name }}', 's$
rvice_type': '{{ item.type }}', 'description': '{{ item.description }}', 'region_name': '{{ service_ks_register_region_name }}', 'au$
h': '{{ service_ks_register_auth }}', 'interface': '{{ service_ks_register_interface }}', 'cacert': '{{ service_ks_cacert }}'}] ***

FAILED - RETRYING: [controllera]: keystone | Creating services (5 retries left).
FAILED - RETRYING: [controllera]: keystone | Creating services (4 retries left).
FAILED - RETRYING: [controllera]: keystone | Creating services (3 retries left).
FAILED - RETRYING: [controllera]: keystone | Creating services (2 retries left).
FAILED - RETRYING: [controllera]: keystone | Creating services (1 retries left).failed: [controllera] (item={'name': 'keystone', 'service_type': 'identity'}) => {"action": "os_keystone_service", "ansible_loop_var"
: "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "admin",
 "url": "https://dashint.example.com:35357"}, {"interface": "internal", "url": "https://dashint.example.com:5000"}, {"interface":
 "public", "url": "https://dash.example.com:5000"}], "name": "keystone", "type": "identity"}, "module_stderr": "Failed to discover
available identity versions when contacting https://dashint.example.com:35357. Attempting to parse version from URL.\nTraceback (mo
st recent call last):\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 710, in urlopen\n    chunk
ed=chunked,\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 386, in _make_request\n    self._val
idate_conn(conn)\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 1040, in _validate_conn\n    co
nn.connect()\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connection.py\", line 426, in connect\n    tls_in_tls=tls_in_
tls,\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 450, in ssl_wrap_socket\n    sock, context, tls_
in_tls, server_hostname=server_hostname\n  File \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_
wrap_socket_impl\n    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n  File \"/usr/lib64/python3.6/ssl.py\",
line 365, in wrap_socket\n    _context=self, _session=session)\n  File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n    se
lf.do_handshake()\n  File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n    self._sslobj.do_handshake()\n  File \"/usr
/lib64/python3.6/ssl.py\", line 648, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED]
 certificate verify failed
(_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most rec
ent call last):\n  File \"/opt/ansible/lib/python3.6/site-packages/requests/adapters.py\", line 450, in send\n    timeout=timeout


I don't know what this task is, the container is running, what does mean service-ks-register : keystone ?

Regards.

Le mar. 15 nov. 2022 à 11:54, Eugen Block <eblock@nde.ag> a écrit :
Okay, I understand. Did you verify if the self-signed cert contains 
everything you require as I wrote in the previous email? Can you paste 
the openssl command output (and mask everything non-public)?

Zitat von wodel youchi <wodel.youchi@gmail.com>:

> Hi,
> Thanks again,
>
> About your question : so with the previous cert it worked but only because
> you had the verification set to false, correct?
> The answer is : Not exactly.
>
> Let me explain, I deployed using a commercial valid certificate, but I
> configured kolla_verify_tls_backend to false exactly to avoid the problem I
> am facing now. From what I have understood :
> kolla_verify_tls_backend=false, means : accept the connection even if the
> verification fails, but apparently it is not the case.
> And kolla_copy_ca_into_containers was positioned to yes from the beginning.
>
> What happened is that my certificate expired, and now I am searching for a
> way to install a self-signed certificate while waiting to get the new
> certificate.
>
> I backported the platform a few days before the expiration of the
> certificate, then I generated the self-signed certificate and I tried to
> deploy it but without success.
>
> Regards.
>
> Le lun. 14 nov. 2022 à 14:21, Eugen Block <eblock@nde.ag> a écrit :
>
>> Hi,
>>
>> > First I want to correct something, the *kolla_verify_tls_backend* was
>> > positioned to *false* from the beginning, while doing the first
>> deployment
>> > with the commercial certificate.
>>
>> so with the previous cert it worked but only because you had the
>> verification set to false, correct?
>>
>> > What do you mean by using openssl? Do you mean to execute the command
>> > inside a container and try to connect to keystone? If yes what is the
>> > correct command?
>>
>> That's one example, yes. Is apache configured correctly to use the
>> provided certs? In my manual deployment it looks like this (only the
>> relevant part):
>>
>> control01:~ # cat /etc/apache2/vhosts.d/keystone-public.conf
>> [...]
>>      SSLEngine On
>>      SSLCertificateFile /etc/ssl/servercerts/control01.fqdn.cert.pem
>>      SSLCACertificateFile /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT
>>      SSLCertificateKeyFile /etc/ssl/private/control01.fqdn.key.pem
>>      SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
>>
>>      # HTTP Strict Transport Security (HSTS) enforces that all
>> communications
>>      # with a server go over SSL. This mitigates the threat from attacks
>> such
>>      # as SSL-Strip which replaces links on the wire, stripping away
>> https prefixes
>>      # and potentially allowing an attacker to view confidential
>> information on the
>>      # wire
>>      Header add Strict-Transport-Security "max-age=15768000"
>> [...]
>>
>> and then test it with:
>>
>> ---snip---
>> control01:~ # curl -v https://control.fqdn:5000/v3
>> [...]
>> * ALPN, offering h2
>> * ALPN, offering http/1.1
>> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>> * TLSv1.3 (IN), TLS handshake, Server hello (2):
>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>> * TLSv1.3 (IN), TLS handshake, Certificate (11):
>> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>> * TLSv1.3 (IN), TLS handshake, Finished (20):
>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>> * TLSv1.3 (OUT), TLS handshake, Finished (20):
>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>> * ALPN, server accepted to use http/1.1
>> * Server certificate:
>> [...]
>> *  subjectAltName: host "control.fqdn" matched cert's "*.fqdn"
>> *  issuer: *******
>> *  SSL certificate verify ok.
>> > GET /v3 HTTP/1.1
>> > Host: control.fqdn:5000
>> > User-Agent: curl/7.66.0
>> > Accept: */*
>> >
>> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>> * old SSL session ID is stale, removing
>> * Mark bundle as not supporting multiuse
>> < HTTP/1.1 200 OK
>> [...]
>> * Connection #0 to host control.fqdn left intact
>> {"version": {"id": "v3.14", "status": "stable", "updated":
>> "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href":
>> "https://control.fqdn:5000/v3/"}], "media-types": [{"base":
>> "application/json", "type":
>> "application/vnd.openstack.identity-v3+json"}]}}
>> ---snip---
>>
>> To check the created certificate you could run something like this:
>>
>> openssl x509 -in /etc/ssl/servercerts/control01.fqdn.cert.pem -text -noout
>>
>> and see if the SANs match your control node(s) IP addresses and FQDNs.
>>
>> Zitat von wodel youchi <wodel.youchi@gmail.com>:
>>
>> > Hi
>> >
>> > Thanks for your help.
>> >
>> > First I want to correct something, the *kolla_verify_tls_backend* was
>> > positioned to *false* from the beginning, while doing the first
>> deployment
>> > with the commercial certificate.
>> >
>> > And yes I have *kolla_copy_ca_into_containers* positioned to *yes* from
>> the
>> > beginning. And I can see in the nodes that there is a directory named
>> > certificates in every module's directory in /etc/kolla
>> >
>> > What do you mean by using openssl? Do you mean to execute the command
>> > inside a container and try to connect to keystone? If yes what is the
>> > correct command?
>> >
>> > It seems like something is missing to tell the client side to ignore the
>> > certificate validity, something like the --insecure parameter in the
>> > openstack cli.
>> >
>> > Regards.
>> >
>> > On Fri, Nov 11, 2022, 21:21 Eugen Block <eblock@nde.ag> wrote:
>> >
>> >> Hi,
>> >>
>> >> I'm not familiar with kolla, but the docs also mention this option:
>> >>
>> >> kolla_copy_ca_into_containers: "yes"
>> >>
>> >> As I understand it the CA cert is required within the containers so
>> >> they can trust the self-signed certs. At least that's how I configure
>> >> it in a manually deployed openstack cloud. Do you have that option
>> >> enabled? If it is enabled, did you verify it with openssl tools?
>> >>
>> >> Regards,
>> >> Eugen
>> >>
>> >> Zitat von wodel youchi <wodel.youchi@gmail.com>:
>> >>
>> >> > Some help please.
>> >> >
>> >> > On Tue, Nov 8, 2022, 14:44 wodel youchi <wodel.youchi@gmail.com>
>> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> To deploy Openstack with a self-signed certificate, the documentation
>> >> says
>> >> >> to generate the certificates using kolla-ansible certificates, to
>> >> configure
>> >> >> the support of TLS in globals.yml and to deploy.
>> >> >>
>> >> >> I am facing a problem, my old certificate has expired, I want to use
>> a
>> >> >> self-signed certificate.
>> >> >> I backported my servers to an older date, then generated a
>> self-signed
>> >> >> certificate using kolla, but the deploy/reconfigure won't work, they
>> >> say :
>> >> >>
>> >> >> self._sslobj.do_handshake()\n  File \"/usr/lib64/python3.6/ssl.py\",
>> >> line
>> >> >> 648, in do_handshakeself._sslobj.do_handshake()\nssl.SSLError: [SSL:
>> >> >> CERTIFICATE_VERIFY_FAILED certificate verify failed
>> >> >>
>> >> >> PS : in my globals.yml i have : *kolla_verify_tls_backend: "yes"*
>> >> >>
>> >> >> Regards.
>> >> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>>
>>
>>
>>