[all] Eventlet broken again with SSL, this time under Python 3.9
noonedeadpunk at ya.ru
Mon Feb 1 12:21:45 UTC 2021
Yes, I can confirm that amqp of version 5.0.3 and later does not accept self-signed certificates in case root ca has not been provided.
It has been bumped to 5.0.5 in u-c recently which made things fail for us everywhere now.
However, in case of adding root CA into the system things continue working properly.
01.02.2021, 11:05, "Alfredo Moralejo Alonso" <amoralej at redhat.com>:
> We updated kombu and amqp on Jan 28th in RDO https://review.rdoproject.org/r/#/c/31661/ so it may be related to it.
> Could you point me to some logs about the failure?
> Best regards.
> On Sat, Jan 30, 2021 at 1:15 PM Dmitriy Rabotyagov <noonedeadpunk at ya.ru> wrote:
>> Yeah, they do:
>> [root at centos-distro openstack-ansible]# rpm -qa | egrep "amqp|kombu"
>> [root at centos-distro openstack-ansible]#
>> But not sure about keystoneauth1 since I see this at the point in oslo.messaging. Full error in systemd looks like this:
>> Jan 30 11:51:04 aio1 nova-conductor: 2021-01-30 11:51:04.543 97314 ERROR oslo.messaging._drivers.impl_rabbit [req-61609624-b577-475d-996e-bc8f9899eae0 - - - - -] Connection failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
>> 30.01.2021, 12:42, "Thomas Goirand" <zigo at debian.org>:
>>> On 1/30/21 10:47 AM, Dmitriy Rabotyagov wrote:
>>>> In the meanwhile we see that most of the services fail to interact with rabbitmq over self-signed SSL in case RDO packages are used even with Python 3.6.
>>>> We don't see this happening when installing things with pip packages though. Both rdo and pip version of eventlet we used was 0.30.0.
>>>> RDO started failing for us several days back with:
>>>> ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
>>>> Not sure, maybe it's not related directly to eventlet, but sounds like it might be.
>>> Does RDO has version 5.0.3 of AMQP and version 5.0.2 of Kombu? That's
>>> what I had to do in Debian to pass this stage.
>>> Though the next issue is what I wrote, when a service tries to validate
>>> a keystone token (ie: keystoneauth1 calls requests that calls urllib3,
>>> which in turns calls Python 3.9 SSL, and then crash with maximum
>>> recursion depth exceeded). I'm no 100% sure the problem is in Eventlet,
>>> but it really looks like it, as it's similar to another SSL crash we had
>>> in Python 3.7.
>>> Thomas Goirand (zigo)
>> Kind Regards,
>> Dmitriy Rabotyagov
More information about the openstack-discuss