We updated kombu and amqp on Jan 28th in RDO https://review.rdoproject.org/r/#/c/31661/ so it may be related to it. Could you point me to some logs about the failure? Best regards. Alfredo On Sat, Jan 30, 2021 at 1:15 PM Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote:
Yeah, they do: [root@centos-distro openstack-ansible]# rpm -qa | egrep "amqp|kombu" python3-kombu-5.0.2-1.el8.noarch python3-amqp-5.0.3-1.el8.noarch [root@centos-distro openstack-ansible]#
But not sure about keystoneauth1 since I see this at the point in oslo.messaging. Full error in systemd looks like this: Jan 30 11:51:04 aio1 nova-conductor[97314]: 2021-01-30 11:51:04.543 97314 ERROR oslo.messaging._drivers.impl_rabbit [req-61609624-b577-475d-996e-bc8f9899eae0 - - - - -] Connection failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
On 1/30/21 10:47 AM, Dmitriy Rabotyagov wrote:
In the meanwhile we see that most of the services fail to interact with rabbitmq over self-signed SSL in case RDO packages are used even with Python 3.6. We don't see this happening when installing things with pip packages
30.01.2021, 12:42, "Thomas Goirand" <zigo@debian.org>: though. Both rdo and pip version of eventlet we used was 0.30.0.
RDO started failing for us several days back with: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed (_ssl.c:897)
Not sure, maybe it's not related directly to eventlet, but sounds like
it might be.
Does RDO has version 5.0.3 of AMQP and version 5.0.2 of Kombu? That's what I had to do in Debian to pass this stage.
Though the next issue is what I wrote, when a service tries to validate a keystone token (ie: keystoneauth1 calls requests that calls urllib3, which in turns calls Python 3.9 SSL, and then crash with maximum recursion depth exceeded). I'm no 100% sure the problem is in Eventlet, but it really looks like it, as it's similar to another SSL crash we had in Python 3.7.
Cheers,
Thomas Goirand (zigo)
-- Kind Regards, Dmitriy Rabotyagov