<div dir="ltr"><div>Hi,</div><div><br></div><div>This is the server certificate generated by kolla<br></div><div><br></div><div> # openssl x509 -noout -text -in <b>backend-cert.pem</b><br>Certificate:<br> Data:<br> Version: 3 (0x2)<br> Serial Number:<br> 36:c4:48:24:e7:88:c4:f0:dd:32:b3:d8:e9:b7:c5:17:5c:4e:85:ff<br> Signature Algorithm: sha256WithRSAEncryption<br> <b>Issuer: CN = KollaTestCA<br> Validity<br> Not Before: Oct 14 13:13:04 2022 GMT<br> Not After : Feb 26 13:13:04 2024 GMT</b><br> Subject: C = US, ST = NC, L = RTP, OU = kolla<br> Subject Public Key Info:<br> Public Key Algorithm: rsaEncryption<br> RSA Public-Key: (2048 bit)<br> Modulus:<br> 00:b9:f6:f9:83:e6:8c:de:fb:3e:6f:df:23:b9:46:<br> 53:04:52:7a:45:44:6e:9b:cb:cc:30:ab:df:bc:b2:</div><div> ....<br> Exponent: 65537 (0x10001)<br> X509v3 extensions:<br> X509v3 Subject Alternative Name:<br> <b> IP Address:20.3.0.23, IP Address:20.3.0.27, IP Address:20.3.0.31</b><br> </div><div><br></div><div>And this is the CA certificate generated by Kolla</div><div># openssl x509 -noout -text -in ca*.pem<br>Certificate:<br> Data:<br> Version: 3 (0x2)<br> Serial Number:<br> 66:c9:c2:c8:fa:45:e7:48:26:a1:48:63:b6:a9:27:1d:dc:74:4a:c3<br> Signature Algorithm: sha256WithRSAEncryption<br><b> Issuer: CN = KollaTestCA<br> Validity<br> Not Before: Oct 14 13:12:59 2022 GMT<br> Not After : Aug 3 13:12:59 2025 GMT<br> Subject: CN = KollaTestCA</b><br> Subject Public Key Info:<br> Public Key Algorithm: rsaEncryption<br> RSA Public-Key: (4096 bit)<br> Modulus:<br> 00:ce:6f:91:5a:bf:81:49:b6:eb:d9:99:60:bc:93:<br> 80:ab:59:bb:20:09:33:b5:b0:75:ba:50:90:87:93:<br></div><div><br></div><div><b><br></b></div><div><b># openssl verify -verbose -CAfile ca.pem backend-cert.pem<br>backend-cert.pem: OK</b></div><div><br></div><div><br></div><div>From the keystone container I got this :</div><div><b>(keystone)[root@controllera /]# curl -v <a href="https://dashint.example.com:5000/v3">https://dashint.example.com:5000/v3</a></b><br>* Trying 20.3.0.1...<br>* TCP_NODELAY set<br>* <b>Connected to <a href="http://dashint.example.com">dashint.example.com</a> (20.3.0.1) port 5000 (#0)</b><br>* ALPN, offering h2<br>* ALPN, offering http/1.1<br><b>* successfully set certificate verify locations:<br>* CAfile: /etc/pki/tls/certs/ca-bundle.crt</b><br> CApath: none<br>* TLSv1.3 (OUT), TLS handshake, Client hello (1):<br>* TLSv1.3 (IN), TLS handshake, Server hello (2):<br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):<br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, Certificate (11):<br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, CERT verify (15):<br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, Finished (20):<br>* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):<br>* TLSv1.3 (OUT), TLS handshake, [no content] (0):<br>* TLSv1.3 (OUT), TLS handshake, Finished (20):<br>* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384<br>* ALPN, server did not agree to a protocol<br>* Server certificate:<br><b>* subject: C=US; ST=NC; L=RTP; OU=kolla<br>* start date: Oct 14 13:13:03 2022 GMT<br>* expire date: Oct 14 13:13:03 2023 GMT<br>* subjectAltName: host "<a href="http://dashint.example.com">dashint.example.com</a>" matched cert's "<a href="http://dashint.example.com">dashint.example.com</a>"</b><br>* issuer: CN=KollaTestCA<br>* SSL certificate verify ok.<br>* TLSv1.3 (OUT), TLS app data, [no content] (0):<br>> GET /v3 HTTP/1.1<br>> Host: <a href="http://dashint.example.com:5000">dashint.example.com:5000</a><br>> User-Agent: curl/7.61.1<br>> Accept: */*<br>><br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):<br>* TLSv1.3 (IN), TLS handshake, [no content] (0):<br>* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):<br>* TLSv1.3 (IN), TLS app data, [no content] (0):<br><b>< HTTP/1.1 200 OK</b><br>< date: Sat, 22 Oct 2022 15:39:22 GMT<br>< server: Apache<br>< content-length: 262<br>< vary: X-Auth-Token<br>< x-openstack-request-id: req-88c293c3-7efb-4a12-ac06-21f90e1fdc10<br>< content-type: application/json<br><<br>* Connection #0 to host <a href="http://dashint.example.com">dashint.example.com</a> left intact<br>{"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "<a href="https://dashint.example.com:5000/v3/">https://dashint.example.com:5000/v3/</a>"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}curl (<a href="https://dashint.example.com:5000/v3">https://dashint.example.com:5000/v3</a>): response: 200, time: 0.012871, size: 262</div><div><br></div><div><br></div><div>When deploying with the self certificate it's in this task on the first controller where the problem is triggered :</div><div><br></div><div><b>TASK [service-ks-register : keystone | Creating services module_name=os_keystone_service, module_args={'name': '{{ <a href="http://item.name">item.name</a> }}', 's$<br>rvice_type': '{{ item.type }}', 'description': '{{ item.description }}', 'region_name': '{{ service_ks_register_region_name }}', 'au$<br>h': '{{ service_ks_register_auth }}', 'interface': '{{ service_ks_register_interface }}', 'cacert': '{{ service_ks_cacert }}'}] ***</b><br>FAILED - RETRYING: [controllera]: keystone | Creating services (5 retries left).<br>FAILED - RETRYING: [controllera]: keystone | Creating services (4 retries left).<br>FAILED - RETRYING: [controllera]: keystone | Creating services (3 retries left).<br>FAILED - RETRYING: [controllera]: keystone | Creating services (2 retries left).<br>FAILED - RETRYING: [controllera]: keystone | Creating services (1 retries left).failed: [controllera] (item={'name': 'keystone', 'service_type': 'identity'}) => {"action": "os_keystone_service", "ansible_loop_var"<br>: "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "admin",<br> "url": "<a href="https://dashint.example.com:35357">https://dashint.example.com:35357</a>"}, {"interface": "internal", "url": "<a href="https://dashint.example.com:5000">https://dashint.example.com:5000</a>"}, {"interface":<br> "public", "url": "<a href="https://dash.example.com:5000">https://dash.example.com:5000</a>"}], "name": "keystone", "type": "identity"}, "module_stderr": "Failed to discover<br>available identity versions when contacting <a href="https://dashint.example.com:35357">https://dashint.example.com:35357</a>. Attempting to parse version from URL.\nTraceback (mo<br>st recent call last):\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 710, in urlopen\n chunk<br>ed=chunked,\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 386, in _make_request\n self._val<br>idate_conn(conn)\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 1040, in _validate_conn\n co<br>nn.connect()\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/connection.py\", line 426, in connect\n tls_in_tls=tls_in_<br>tls,\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 450, in ssl_wrap_socket\n sock, context, tls_<br>in_tls, server_hostname=server_hostname\n File \"/opt/ansible/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_<br>wrap_socket_impl\n return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\",<br>line 365, in wrap_socket\n _context=self, _session=session)\n File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n se<br>lf.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n self._sslobj.do_handshake()\n File \"/usr<br>/lib64/python3.6/ssl.py\", line 648, in do_handshake\n <b>self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED]<br> certificate verify failed</b> (_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most rec<br>ent call last):\n File \"/opt/ansible/lib/python3.6/site-packages/requests/adapters.py\", line 450, in send\n timeout=timeout</div><div><br></div><div><br></div><div>I don't know what this task is, the container is running, what does mean
<b>service-ks-register : keystone</b> ?<br></div><div><br></div><div>Regards.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 15 nov. 2022 à 11:54, Eugen Block <<a href="mailto:eblock@nde.ag">eblock@nde.ag</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Okay, I understand. Did you verify if the self-signed cert contains <br>
everything you require as I wrote in the previous email? Can you paste <br>
the openssl command output (and mask everything non-public)?<br>
<br>
Zitat von wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>>:<br>
<br>
> Hi,<br>
> Thanks again,<br>
><br>
> About your question : so with the previous cert it worked but only because<br>
> you had the verification set to false, correct?<br>
> The answer is : Not exactly.<br>
><br>
> Let me explain, I deployed using a commercial valid certificate, but I<br>
> configured kolla_verify_tls_backend to false exactly to avoid the problem I<br>
> am facing now. From what I have understood :<br>
> kolla_verify_tls_backend=false, means : accept the connection even if the<br>
> verification fails, but apparently it is not the case.<br>
> And kolla_copy_ca_into_containers was positioned to yes from the beginning.<br>
><br>
> What happened is that my certificate expired, and now I am searching for a<br>
> way to install a self-signed certificate while waiting to get the new<br>
> certificate.<br>
><br>
> I backported the platform a few days before the expiration of the<br>
> certificate, then I generated the self-signed certificate and I tried to<br>
> deploy it but without success.<br>
><br>
> Regards.<br>
><br>
> Le lun. 14 nov. 2022 à 14:21, Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> a écrit :<br>
><br>
>> Hi,<br>
>><br>
>> > First I want to correct something, the *kolla_verify_tls_backend* was<br>
>> > positioned to *false* from the beginning, while doing the first<br>
>> deployment<br>
>> > with the commercial certificate.<br>
>><br>
>> so with the previous cert it worked but only because you had the<br>
>> verification set to false, correct?<br>
>><br>
>> > What do you mean by using openssl? Do you mean to execute the command<br>
>> > inside a container and try to connect to keystone? If yes what is the<br>
>> > correct command?<br>
>><br>
>> That's one example, yes. Is apache configured correctly to use the<br>
>> provided certs? In my manual deployment it looks like this (only the<br>
>> relevant part):<br>
>><br>
>> control01:~ # cat /etc/apache2/vhosts.d/keystone-public.conf<br>
>> [...]<br>
>> SSLEngine On<br>
>> SSLCertificateFile /etc/ssl/servercerts/control01.fqdn.cert.pem<br>
>> SSLCACertificateFile /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT<br>
>> SSLCertificateKeyFile /etc/ssl/private/control01.fqdn.key.pem<br>
>> SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown<br>
>><br>
>> # HTTP Strict Transport Security (HSTS) enforces that all<br>
>> communications<br>
>> # with a server go over SSL. This mitigates the threat from attacks<br>
>> such<br>
>> # as SSL-Strip which replaces links on the wire, stripping away<br>
>> https prefixes<br>
>> # and potentially allowing an attacker to view confidential<br>
>> information on the<br>
>> # wire<br>
>> Header add Strict-Transport-Security "max-age=15768000"<br>
>> [...]<br>
>><br>
>> and then test it with:<br>
>><br>
>> ---snip---<br>
>> control01:~ # curl -v <a href="https://control.fqdn:5000/v3" rel="noreferrer" target="_blank">https://control.fqdn:5000/v3</a><br>
>> [...]<br>
>> * ALPN, offering h2<br>
>> * ALPN, offering http/1.1<br>
>> * TLSv1.3 (OUT), TLS handshake, Client hello (1):<br>
>> * TLSv1.3 (IN), TLS handshake, Server hello (2):<br>
>> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):<br>
>> * TLSv1.3 (IN), TLS handshake, Certificate (11):<br>
>> * TLSv1.3 (IN), TLS handshake, CERT verify (15):<br>
>> * TLSv1.3 (IN), TLS handshake, Finished (20):<br>
>> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):<br>
>> * TLSv1.3 (OUT), TLS handshake, Finished (20):<br>
>> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384<br>
>> * ALPN, server accepted to use http/1.1<br>
>> * Server certificate:<br>
>> [...]<br>
>> * subjectAltName: host "control.fqdn" matched cert's "*.fqdn"<br>
>> * issuer: *******<br>
>> * SSL certificate verify ok.<br>
>> > GET /v3 HTTP/1.1<br>
>> > Host: control.fqdn:5000<br>
>> > User-Agent: curl/7.66.0<br>
>> > Accept: */*<br>
>> ><br>
>> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):<br>
>> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):<br>
>> * old SSL session ID is stale, removing<br>
>> * Mark bundle as not supporting multiuse<br>
>> < HTTP/1.1 200 OK<br>
>> [...]<br>
>> * Connection #0 to host control.fqdn left intact<br>
>> {"version": {"id": "v3.14", "status": "stable", "updated":<br>
>> "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href":<br>
>> "<a href="https://control.fqdn:5000/v3/" rel="noreferrer" target="_blank">https://control.fqdn:5000/v3/</a>"}], "media-types": [{"base":<br>
>> "application/json", "type":<br>
>> "application/vnd.openstack.identity-v3+json"}]}}<br>
>> ---snip---<br>
>><br>
>> To check the created certificate you could run something like this:<br>
>><br>
>> openssl x509 -in /etc/ssl/servercerts/control01.fqdn.cert.pem -text -noout<br>
>><br>
>> and see if the SANs match your control node(s) IP addresses and FQDNs.<br>
>><br>
>> Zitat von wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>>:<br>
>><br>
>> > Hi<br>
>> ><br>
>> > Thanks for your help.<br>
>> ><br>
>> > First I want to correct something, the *kolla_verify_tls_backend* was<br>
>> > positioned to *false* from the beginning, while doing the first<br>
>> deployment<br>
>> > with the commercial certificate.<br>
>> ><br>
>> > And yes I have *kolla_copy_ca_into_containers* positioned to *yes* from<br>
>> the<br>
>> > beginning. And I can see in the nodes that there is a directory named<br>
>> > certificates in every module's directory in /etc/kolla<br>
>> ><br>
>> > What do you mean by using openssl? Do you mean to execute the command<br>
>> > inside a container and try to connect to keystone? If yes what is the<br>
>> > correct command?<br>
>> ><br>
>> > It seems like something is missing to tell the client side to ignore the<br>
>> > certificate validity, something like the --insecure parameter in the<br>
>> > openstack cli.<br>
>> ><br>
>> > Regards.<br>
>> ><br>
>> > On Fri, Nov 11, 2022, 21:21 Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
>> ><br>
>> >> Hi,<br>
>> >><br>
>> >> I'm not familiar with kolla, but the docs also mention this option:<br>
>> >><br>
>> >> kolla_copy_ca_into_containers: "yes"<br>
>> >><br>
>> >> As I understand it the CA cert is required within the containers so<br>
>> >> they can trust the self-signed certs. At least that's how I configure<br>
>> >> it in a manually deployed openstack cloud. Do you have that option<br>
>> >> enabled? If it is enabled, did you verify it with openssl tools?<br>
>> >><br>
>> >> Regards,<br>
>> >> Eugen<br>
>> >><br>
>> >> Zitat von wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>>:<br>
>> >><br>
>> >> > Some help please.<br>
>> >> ><br>
>> >> > On Tue, Nov 8, 2022, 14:44 wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>><br>
>> wrote:<br>
>> >> ><br>
>> >> >> Hi,<br>
>> >> >><br>
>> >> >> To deploy Openstack with a self-signed certificate, the documentation<br>
>> >> says<br>
>> >> >> to generate the certificates using kolla-ansible certificates, to<br>
>> >> configure<br>
>> >> >> the support of TLS in globals.yml and to deploy.<br>
>> >> >><br>
>> >> >> I am facing a problem, my old certificate has expired, I want to use<br>
>> a<br>
>> >> >> self-signed certificate.<br>
>> >> >> I backported my servers to an older date, then generated a<br>
>> self-signed<br>
>> >> >> certificate using kolla, but the deploy/reconfigure won't work, they<br>
>> >> say :<br>
>> >> >><br>
>> >> >> self._sslobj.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\",<br>
>> >> line<br>
>> >> >> 648, in do_handshakeself._sslobj.do_handshake()\nssl.SSLError: [SSL:<br>
>> >> >> CERTIFICATE_VERIFY_FAILED certificate verify failed<br>
>> >> >><br>
>> >> >> PS : in my globals.yml i have : *kolla_verify_tls_backend: "yes"*<br>
>> >> >><br>
>> >> >> Regards.<br>
>> >> >><br>
>> >><br>
>> >><br>
>> >><br>
>> >><br>
>> >><br>
>><br>
>><br>
>><br>
>><br>
<br>
<br>
<br>
<br>
</blockquote></div>