Yes, I can confirm that amqp of version 5.0.3 and later does not accept self-signed certificates in case root ca has not been provided. It has been bumped to 5.0.5 in u-c recently which made things fail for us everywhere now. However, in case of adding root CA into the system things continue working properly. 01.02.2021, 11:05, "Alfredo Moralejo Alonso" <amoralej@redhat.com>:
We updated kombu and amqp on Jan 28th in RDO https://review.rdoproject.org/r/#/c/31661/ so it may be related to it.
Could you point me to some logs about the failure?
Best regards.
Alfredo
On Sat, Jan 30, 2021 at 1:15 PM Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote:
Yeah, they do: [root@centos-distro openstack-ansible]# rpm -qa | egrep "amqp|kombu" python3-kombu-5.0.2-1.el8.noarch python3-amqp-5.0.3-1.el8.noarch [root@centos-distro openstack-ansible]#
But not sure about keystoneauth1 since I see this at the point in oslo.messaging. Full error in systemd looks like this: Jan 30 11:51:04 aio1 nova-conductor[97314]: 2021-01-30 11:51:04.543 97314 ERROR oslo.messaging._drivers.impl_rabbit [req-61609624-b577-475d-996e-bc8f9899eae0 - - - - -] Connection failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
30.01.2021, 12:42, "Thomas Goirand" <zigo@debian.org>:
On 1/30/21 10:47 AM, Dmitriy Rabotyagov wrote:
In the meanwhile we see that most of the services fail to interact with rabbitmq over self-signed SSL in case RDO packages are used even with Python 3.6. We don't see this happening when installing things with pip packages though. Both rdo and pip version of eventlet we used was 0.30.0.
RDO started failing for us several days back with: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
Not sure, maybe it's not related directly to eventlet, but sounds like it might be.
Does RDO has version 5.0.3 of AMQP and version 5.0.2 of Kombu? That's what I had to do in Debian to pass this stage.
Though the next issue is what I wrote, when a service tries to validate a keystone token (ie: keystoneauth1 calls requests that calls urllib3, which in turns calls Python 3.9 SSL, and then crash with maximum recursion depth exceeded). I'm no 100% sure the problem is in Eventlet, but it really looks like it, as it's similar to another SSL crash we had in Python 3.7.
Cheers,
Thomas Goirand (zigo)
-- Kind Regards, Dmitriy Rabotyagov
-- Kind Regards, Dmitriy Rabotyagov