[kolla-ansible][letsencrypt] containers are running but not getting certificates.
Hi All, I'm trying to do a test multinode deploy using 2023.2 I have letsencrypt_webserver and letsencrypt_lego contsainers running and I'm seeing random traffic in the /var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly confident they're plumbed through to the public internet properly, but I don't seem to be getting certificates. how can I trigger a renewal attempt so I can maybe see what I've screwed up? Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up? Of course as soon as I ask I find the answer and more questions. `exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. What my `kolla-ansible deploy` is actually dying on is currently: fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."} so perhaps there's something I need ot turn "off" in `globals.yml`? -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Hi, Can u send me content of /etc/kolla ? And also config in globals regarding tls ? Kevko Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline> po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx <jon@csail.mit.edu> napsal:
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up?
Of course as soon as I ask I find the answer and more questions.
`exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed.
What my `kolla-ansible deploy` is actually dying on is currently:
fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."}
so perhaps there's something I need ot turn "off" in `globals.yml`?
-- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Btw, did you follow docs ? Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline> út 21. 5. 2024 v 21:03 odesílatel Michal Arbet <michal.arbet@ultimum.io> napsal:
Hi,
Can u send me content of /etc/kolla ?
And also config in globals regarding tls ?
Kevko Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io <https://ultimum.io/>*
LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>
po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx <jon@csail.mit.edu> napsal:
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up?
Of course as soon as I ask I find the answer and more questions.
`exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed.
What my `kolla-ansible deploy` is actually dying on is currently:
fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."}
so perhaps there's something I need ot turn "off" in `globals.yml`?
-- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
On Tue, May 21, 2024 at 09:04:23PM +0200, Michal Arbet wrote: :Btw, did you follow docs ? been reading https://docs.openstack.org/kolla-ansible/2023.2/admin/tls.html I'm a bit unclear which sections apply with letsencrypt info and which it replaces (probably the config snip I sent will show my possibly flawed understanding). -Jon :Michal Arbet :Openstack Engineer : :Ultimum Technologies a.s. :Na Poříčí 1047/26, 11000 Praha 1 :Czech Republic : :+420 604 228 897 :michal.arbet@ultimum.io :*https://ultimum.io <https://ultimum.io/>* : :LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter :<https://twitter.com/ultimumtech> | Facebook :<https://www.facebook.com/ultimumtechnologies/timeline> : : :út 21. 5. 2024 v 21:03 odesílatel Michal Arbet <michal.arbet@ultimum.io> :napsal: : :> Hi, :> :> Can u send me content of /etc/kolla ? :> :> And also config in globals regarding tls ? :> :> Kevko :> Michal Arbet :> Openstack Engineer :> :> Ultimum Technologies a.s. :> Na Poříčí 1047/26, 11000 Praha 1 :> Czech Republic :> :> +420 604 228 897 :> michal.arbet@ultimum.io :> *https://ultimum.io <https://ultimum.io/>* :> :> LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter :> <https://twitter.com/ultimumtech> | Facebook :> <https://www.facebook.com/ultimumtechnologies/timeline> :> :> :> po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx <jon@csail.mit.edu> :> napsal: :> :>> On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :>> :Hi All, :>> : :>> :I'm trying to do a test multinode deploy using 2023.2 :>> : :>> :I have letsencrypt_webserver and letsencrypt_lego contsainers running :>> :and I'm seeing random traffic in the :>> :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :>> :confident they're plumbed through to the public internet properly, but :>> :I don't seem to be getting certificates. :>> : :>> :how can I trigger a renewal attempt so I can maybe see what I've :>> :screwed up? :>> :>> Of course as soon as I ask I find the answer and more questions. :>> :>> `exec`ing the /usr/bin/letsencrypt-certificates line from :>> `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego :>> container does get a letsencrypt cert into th haproxy container as :>> `/etc/haproxy/certificates/haproxy-internal.pem` however there's also :>> a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. :>> :>> :>> What my `kolla-ansible deploy` is actually dying on is currently: :>> :>> fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred :>> while templating '{{ lookup('first_found', certs) }}'. Error was a <class :>> 'ansible.errors.AnsibleLookupError'>, original message: No file was found :>> when using first_found."} :>> :>> so perhaps there's something I need ot turn "off" in `globals.yml`? :>> :>> :>> -- :>> Jonathan Proulx (he/him) :>> Sr. Technical Architect :>> The Infrastructure Group :>> MIT CSAIL :>> :> -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Running into somewhat of the same issues, This use case is very badly documented currently, I have tested this deployment under 2024.1 I have found you need the following so far kolla_enable_tls_external: "yes" letsencrypt_email: "xxx" enable_letsencrypt: yes letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" # attempt to renew Let's Encrypt certificate every 12 hours letsencrypt_cron_renew_schedule: "0 */12 * * *" All other TLS configuration options should be left to their defaults A good sign that this worked is to check /var/log/kolla/letsencrypt/letsencrypt-lego.log [/etc/haproxy/certificates/haproxy.pem - update] Transaction /var/lib/haproxy/haproxy.pem -> /etc/haproxy/certificates/haproxy.pem successful. ________________________________ From: Jonathan Proulx <jon@csail.mit.edu> Sent: Wednesday, May 22, 2024 9:03 AM To: Michal Arbet <michal.arbet@ultimum.io> Cc: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [kolla-ansible][letsencrypt] containers are running but not getting certificates. On Tue, May 21, 2024 at 09:04:23PM +0200, Michal Arbet wrote: :Btw, did you follow docs ? been reading https://docs.openstack.org/kolla-ansible/2023.2/admin/tls.html I'm a bit unclear which sections apply with letsencrypt info and which it replaces (probably the config snip I sent will show my possibly flawed understanding). -Jon :Michal Arbet :Openstack Engineer : :Ultimum Technologies a.s. :Na Poříčí 1047/26, 11000 Praha 1 :Czech Republic : :+420 604 228 897 :michal.arbet@ultimum.io :*https://ultimum.io <https://ultimum.io/>* : :LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter :<https://twitter.com/ultimumtech> | Facebook :<https://www.facebook.com/ultimumtechnologies/timeline> : : :út 21. 5. 2024 v 21:03 odesílatel Michal Arbet <michal.arbet@ultimum.io> :napsal: : :> Hi, :> :> Can u send me content of /etc/kolla ? :> :> And also config in globals regarding tls ? :> :> Kevko :> Michal Arbet :> Openstack Engineer :> :> Ultimum Technologies a.s. :> Na Poříčí 1047/26, 11000 Praha 1 :> Czech Republic :> :> +420 604 228 897 :> michal.arbet@ultimum.io :> *https://ultimum.io <https://ultimum.io/>* :> :> LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter :> <https://twitter.com/ultimumtech> | Facebook :> <https://www.facebook.com/ultimumtechnologies/timeline> :> :> :> po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx <jon@csail.mit.edu> :> napsal: :> :>> On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :>> :Hi All, :>> : :>> :I'm trying to do a test multinode deploy using 2023.2 :>> : :>> :I have letsencrypt_webserver and letsencrypt_lego contsainers running :>> :and I'm seeing random traffic in the :>> :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :>> :confident they're plumbed through to the public internet properly, but :>> :I don't seem to be getting certificates. :>> : :>> :how can I trigger a renewal attempt so I can maybe see what I've :>> :screwed up? :>> :>> Of course as soon as I ask I find the answer and more questions. :>> :>> `exec`ing the /usr/bin/letsencrypt-certificates line from :>> `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego :>> container does get a letsencrypt cert into th haproxy container as :>> `/etc/haproxy/certificates/haproxy-internal.pem` however there's also :>> a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. :>> :>> :>> What my `kolla-ansible deploy` is actually dying on is currently: :>> :>> fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred :>> while templating '{{ lookup('first_found', certs) }}'. Error was a <class :>> 'ansible.errors.AnsibleLookupError'>, original message: No file was found :>> when using first_found."} :>> :>> so perhaps there's something I need ot turn "off" in `globals.yml`? :>> :>> :>> -- :>> Jonathan Proulx (he/him) :>> Sr. Technical Architect :>> The Infrastructure Group :>> MIT CSAIL :>> :> -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
On Tue, May 21, 2024 at 09:03:56PM +0200, Michal Arbet wrote: :Hi, : :Can u send me content of /etc/kolla ? root@kolla:~# ls -lR /etc/kolla /etc/kolla: total 188 -rw-rw-r-- 1 root root 33375 May 20 14:23 globals.yml -rw-rw-r-- 1 root root 33343 May 20 12:37 globals.yml~ -rw-r--r-- 1 root root 8999 May 20 12:26 hosts -rw-r--r-- 1 root root 8999 May 20 12:24 hosts~ -rw-rw-r-- 1 root root 194 May 16 14:14 kolla-build.conf -rw-rw-r-- 1 root root 156 May 16 13:43 kolla-build.conf~ -rw-r----- 1 root root 38554 May 16 15:30 passwords.yml -rw-r----- 1 root root 38507 May 16 15:21 passwords.yml~ -rw-r--r-- 1 root root 2105 May 16 13:39 sources.list :And also config in globals regarding tls ? root@kolla:~# grep -e tls -e acme -e letsencrypt /etc/kolla/globals.yml #om_enable_rabbitmq_tls: "{{ rabbitmq_enable_tls | bool }}" kolla_enable_tls_internal: "yes" #kolla_enable_tls_external: "{{ kolla_enable_tls_internal if kolla_same_external_internal_vip | bool else 'no' }}" #kolla_enable_tls_backend: "no" kolla_enable_tls_backend: "yes" #kolla_verify_tls_backend: "yes" #kolla_tls_backend_cert: "{{ kolla_certificates_dir }}/backend-cert.pem" #kolla_tls_backend_key: "{{ kolla_certificates_dir }}/backend-key.pem" #acme_client_servers: enable_letsencrypt: "yes" # This option is required for letsencrypt role to work properly. letsencrypt_email: "redacted@valid.domain.edu" #letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" #letsencrypt_cron_renew_schedule: "0 */12 * * *" #rabbitmq_enable_tls: "no" Thanks, -Jon : :Kevko :Michal Arbet :Openstack Engineer : :Ultimum Technologies a.s. :Na Poříčí 1047/26, 11000 Praha 1 :Czech Republic : :+420 604 228 897 :michal.arbet@ultimum.io :*https://ultimum.io <https://ultimum.io/>* : :LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter :<https://twitter.com/ultimumtech> | Facebook :<https://www.facebook.com/ultimumtechnologies/timeline> : : :po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx <jon@csail.mit.edu> :napsal: : :> On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :> :Hi All, :> : :> :I'm trying to do a test multinode deploy using 2023.2 :> : :> :I have letsencrypt_webserver and letsencrypt_lego contsainers running :> :and I'm seeing random traffic in the :> :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :> :confident they're plumbed through to the public internet properly, but :> :I don't seem to be getting certificates. :> : :> :how can I trigger a renewal attempt so I can maybe see what I've :> :screwed up? :> :> Of course as soon as I ask I find the answer and more questions. :> :> `exec`ing the /usr/bin/letsencrypt-certificates line from :> `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego :> container does get a letsencrypt cert into th haproxy container as :> `/etc/haproxy/certificates/haproxy-internal.pem` however there's also :> a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. :> :> :> What my `kolla-ansible deploy` is actually dying on is currently: :> :> fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred :> while templating '{{ lookup('first_found', certs) }}'. Error was a <class :> 'ansible.errors.AnsibleLookupError'>, original message: No file was found :> when using first_found."} :> :> so perhaps there's something I need ot turn "off" in `globals.yml`? :> :> :> -- :> Jonathan Proulx (he/him) :> Sr. Technical Architect :> The Infrastructure Group :> MIT CSAIL :> -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
participants (3)
-
Forrest Fuqua
-
Jonathan Proulx
-
Michal Arbet