[kolla-ansible][letsencrypt] containers are running but not getting certificates.
Hi All,
I'm trying to do a test multinode deploy using 2023.2
I have letsencrypt_webserver and letsencrypt_lego contsainers running and I'm seeing random traffic in the /var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly confident they're plumbed through to the public internet properly, but I don't seem to be getting certificates.
how can I trigger a renewal attempt so I can maybe see what I've screwed up?
Thanks, -Jon
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up?
Of course as soon as I ask I find the answer and more questions.
`exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed.
What my `kolla-ansible deploy` is actually dying on is currently:
fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."}
so perhaps there's something I need ot turn "off" in `globals.yml`?
Hi,
Can u send me content of /etc/kolla ?
And also config in globals regarding tls ?
Kevko Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx jon@csail.mit.edu napsal:
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up?
Of course as soon as I ask I find the answer and more questions.
`exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed.
What my `kolla-ansible deploy` is actually dying on is currently:
fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."}
so perhaps there's something I need ot turn "off" in `globals.yml`?
-- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Btw, did you follow docs ? Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
út 21. 5. 2024 v 21:03 odesílatel Michal Arbet michal.arbet@ultimum.io napsal:
Hi,
Can u send me content of /etc/kolla ?
And also config in globals regarding tls ?
Kevko Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx jon@csail.mit.edu napsal:
On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to do a test multinode deploy using 2023.2 : :I have letsencrypt_webserver and letsencrypt_lego contsainers running :and I'm seeing random traffic in the :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :confident they're plumbed through to the public internet properly, but :I don't seem to be getting certificates. : :how can I trigger a renewal attempt so I can maybe see what I've :screwed up?
Of course as soon as I ask I find the answer and more questions.
`exec`ing the /usr/bin/letsencrypt-certificates line from `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego container does get a letsencrypt cert into th haproxy container as `/etc/haproxy/certificates/haproxy-internal.pem` however there's also a `/etc/haproxy/certificates/haproxy.pem` that is self-signed.
What my `kolla-ansible deploy` is actually dying on is currently:
fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred while templating '{{ lookup('first_found', certs) }}'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found."}
so perhaps there's something I need ot turn "off" in `globals.yml`?
-- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
On Tue, May 21, 2024 at 09:04:23PM +0200, Michal Arbet wrote: :Btw, did you follow docs ?
been reading https://docs.openstack.org/kolla-ansible/2023.2/admin/tls.html
I'm a bit unclear which sections apply with letsencrypt info and which it replaces (probably the config snip I sent will show my possibly flawed understanding).
-Jon
:Michal Arbet :Openstack Engineer : :Ultimum Technologies a.s. :Na Poříčí 1047/26, 11000 Praha 1 :Czech Republic : :+420 604 228 897 :michal.arbet@ultimum.io :*https://ultimum.io https://ultimum.io/* : :LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter :https://twitter.com/ultimumtech | Facebook :https://www.facebook.com/ultimumtechnologies/timeline : : :út 21. 5. 2024 v 21:03 odesílatel Michal Arbet michal.arbet@ultimum.io :napsal: : :> Hi, :> :> Can u send me content of /etc/kolla ? :> :> And also config in globals regarding tls ? :> :> Kevko :> Michal Arbet :> Openstack Engineer :> :> Ultimum Technologies a.s. :> Na Poříčí 1047/26, 11000 Praha 1 :> Czech Republic :> :> +420 604 228 897 :> michal.arbet@ultimum.io :> *https://ultimum.io https://ultimum.io/* :> :> LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter :> https://twitter.com/ultimumtech | Facebook :> https://www.facebook.com/ultimumtechnologies/timeline :> :> :> po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx jon@csail.mit.edu :> napsal: :> :>> On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :>> :Hi All, :>> : :>> :I'm trying to do a test multinode deploy using 2023.2 :>> : :>> :I have letsencrypt_webserver and letsencrypt_lego contsainers running :>> :and I'm seeing random traffic in the :>> :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :>> :confident they're plumbed through to the public internet properly, but :>> :I don't seem to be getting certificates. :>> : :>> :how can I trigger a renewal attempt so I can maybe see what I've :>> :screwed up? :>> :>> Of course as soon as I ask I find the answer and more questions. :>> :>> `exec`ing the /usr/bin/letsencrypt-certificates line from :>> `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego :>> container does get a letsencrypt cert into th haproxy container as :>> `/etc/haproxy/certificates/haproxy-internal.pem` however there's also :>> a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. :>> :>> :>> What my `kolla-ansible deploy` is actually dying on is currently: :>> :>> fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred :>> while templating '{{ lookup('first_found', certs) }}'. Error was a <class :>> 'ansible.errors.AnsibleLookupError'>, original message: No file was found :>> when using first_found."} :>> :>> so perhaps there's something I need ot turn "off" in `globals.yml`? :>> :>> :>> -- :>> Jonathan Proulx (he/him) :>> Sr. Technical Architect :>> The Infrastructure Group :>> MIT CSAIL :>> :>
On Tue, May 21, 2024 at 09:03:56PM +0200, Michal Arbet wrote: :Hi, : :Can u send me content of /etc/kolla ?
root@kolla:~# ls -lR /etc/kolla /etc/kolla: total 188 -rw-rw-r-- 1 root root 33375 May 20 14:23 globals.yml -rw-rw-r-- 1 root root 33343 May 20 12:37 globals.yml~ -rw-r--r-- 1 root root 8999 May 20 12:26 hosts -rw-r--r-- 1 root root 8999 May 20 12:24 hosts~ -rw-rw-r-- 1 root root 194 May 16 14:14 kolla-build.conf -rw-rw-r-- 1 root root 156 May 16 13:43 kolla-build.conf~ -rw-r----- 1 root root 38554 May 16 15:30 passwords.yml -rw-r----- 1 root root 38507 May 16 15:21 passwords.yml~ -rw-r--r-- 1 root root 2105 May 16 13:39 sources.list
:And also config in globals regarding tls ?
root@kolla:~# grep -e tls -e acme -e letsencrypt /etc/kolla/globals.yml #om_enable_rabbitmq_tls: "{{ rabbitmq_enable_tls | bool }}" kolla_enable_tls_internal: "yes" #kolla_enable_tls_external: "{{ kolla_enable_tls_internal if kolla_same_external_internal_vip | bool else 'no' }}" #kolla_enable_tls_backend: "no" kolla_enable_tls_backend: "yes" #kolla_verify_tls_backend: "yes" #kolla_tls_backend_cert: "{{ kolla_certificates_dir }}/backend-cert.pem" #kolla_tls_backend_key: "{{ kolla_certificates_dir }}/backend-key.pem" #acme_client_servers: enable_letsencrypt: "yes" # This option is required for letsencrypt role to work properly. letsencrypt_email: "redacted@valid.domain.edu" #letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" #letsencrypt_cron_renew_schedule: "0 */12 * * *" #rabbitmq_enable_tls: "no"
Thanks, -Jon
: :Kevko :Michal Arbet :Openstack Engineer : :Ultimum Technologies a.s. :Na Poříčí 1047/26, 11000 Praha 1 :Czech Republic : :+420 604 228 897 :michal.arbet@ultimum.io :*https://ultimum.io https://ultimum.io/* : :LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter :https://twitter.com/ultimumtech | Facebook :https://www.facebook.com/ultimumtechnologies/timeline : : :po 20. 5. 2024 v 22:23 odesílatel Jonathan Proulx jon@csail.mit.edu :napsal: : :> On Mon, May 20, 2024 at 01:44:24PM -0400, Jonathan Proulx wrote: :> :Hi All, :> : :> :I'm trying to do a test multinode deploy using 2023.2 :> : :> :I have letsencrypt_webserver and letsencrypt_lego contsainers running :> :and I'm seeing random traffic in the :> :/var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly :> :confident they're plumbed through to the public internet properly, but :> :I don't seem to be getting certificates. :> : :> :how can I trigger a renewal attempt so I can maybe see what I've :> :screwed up? :> :> Of course as soon as I ask I find the answer and more questions. :> :> `exec`ing the /usr/bin/letsencrypt-certificates line from :> `/usr/local/bin/letsencrypt-lego-run.sh` in the letsencrypt_lego :> container does get a letsencrypt cert into th haproxy container as :> `/etc/haproxy/certificates/haproxy-internal.pem` however there's also :> a `/etc/haproxy/certificates/haproxy.pem` that is self-signed. :> :> :> What my `kolla-ansible deploy` is actually dying on is currently: :> :> fatal: [control0]: FAILED! => {"msg": "An unhandled exception occurred :> while templating '{{ lookup('first_found', certs) }}'. Error was a <class :> 'ansible.errors.AnsibleLookupError'>, original message: No file was found :> when using first_found."} :> :> so perhaps there's something I need ot turn "off" in `globals.yml`? :> :> :> -- :> Jonathan Proulx (he/him) :> Sr. Technical Architect :> The Infrastructure Group :> MIT CSAIL :>
participants (2)
-
Jonathan Proulx
-
Michal Arbet