[all] [ironic] Heads-up: another eventlet-related issue on Stream 8
Hi stackers,
I've noticed this some time ago, but somehow decided that it wouldn't affect OpenStack upstream. Early sign of burn-out? Anyway...
Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a way that breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486). It manifests in the following traceback:
Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected
During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py", line 173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File "/usr/lib/python3.6/site-packages/oslo_service/sslutils.py", line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout'
The fix should be rather simple IMO: just remove the condition on Python 2 from here: https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108. But I'm not sure, given the state of eventlet community, we can land anything these.
The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use a non-default Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold...
Thoughts, ideas?
Dmitry
Isn't this a bug in RedHat's distribution of Python? This seems like the kind of regression that would be guaranteed against in an LTS distribution. Can we get it addressed in that manner?
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 6:10 AM Dmitry Tantsur dtantsur@protonmail.com wrote:
Hi stackers,
I've noticed this some time ago, but somehow decided that it wouldn't affect OpenStack upstream. Early sign of burn-out? Anyway...
Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a way that breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486). It manifests in the following traceback:
Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected
During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py", line 173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File "/usr/lib/python3.6/site-packages/oslo_service/sslutils.py", line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout'
The fix should be rather simple IMO: just remove the condition on Python 2 from here: https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108.
But I'm not sure, given the state of eventlet community, we can land anything these.
The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use a non-default Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold...
Thoughts, ideas?
Dmitry
The thing is, eventlet is hacking into the Python internals. They are not guaranteed to never change. I did raise this problem to the Python team, and basically they consider fixing the CVE more important than keeping weird hacks working.
Dmitry
On 12/8/23 17:09, Jay Faulkner wrote:
Isn't this a bug in RedHat's distribution of Python? This seems like the kind of regression that would be guaranteed against in an LTS distribution. Can we get it addressed in that manner?
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 6:10 AM Dmitry Tantsur <dtantsur@protonmail.com mailto:dtantsur@protonmail.com> wrote:
Hi stackers, I've noticed this some time ago, but somehow decided that it wouldn't affect OpenStack upstream. Early sign of burn-out? Anyway... Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a way that breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486 <https://issues.redhat.com/browse/OCPBUGS-20486>). It manifests in the following traceback: Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py", line 173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File "/usr/lib/python3.6/site-packages/oslo_service/sslutils.py", line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout' The fix should be rather simple IMO: just remove the condition on Python 2 from here: https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108 <https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108>. But I'm not sure, given the state of eventlet community, we can land anything these. The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use a non-default Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold... Thoughts, ideas? Dmitry
There's no path for fixing that in eventlet upstream; which really means there's not a path for fixing this in OpenStack upstream. I suspect this means that we're just incompatible with python 3.6 as RHEL ships it unless RHEL also patches the eventlet shipped to fix compatibility.
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 9:09 AM Dmitry Tantsur dtantsur@protonmail.com wrote:
The thing is, eventlet is hacking into the Python internals. They are not guaranteed to never change. I did raise this problem to the Python team, and basically they consider fixing the CVE more important than keeping weird hacks working.
Dmitry
On 12/8/23 17:09, Jay Faulkner wrote:
Isn't this a bug in RedHat's distribution of Python? This seems like the kind of regression that would be guaranteed against in an LTS distribution. Can we get it addressed in that manner?
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 6:10 AM Dmitry Tantsur <dtantsur@protonmail.com mailto:dtantsur@protonmail.com> wrote:
Hi stackers, I've noticed this some time ago, but somehow decided that it wouldn't affect OpenStack upstream. Early sign of burn-out? Anyway... Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a way that breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486 <https://issues.redhat.com/browse/OCPBUGS-20486>). It manifests inthe
following traceback: Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py",line
173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File "/usr/lib/python3.6/site-packages/oslo_service/sslutils.py", line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout' The fix should be rather simple IMO: just remove the condition on Python 2 from here:https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108 < https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108
. But I'm not sure, given the state of eventlet community, we can land anything these.
The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use anon-default
Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold... Thoughts, ideas? Dmitry
I talked to Dmitry in IRC about this, patching the RPM in RHEL is done; but it's not helpful in this case as IPA ramdisks are built with pypi-copies of eventlet.
We talked about a potential solution of using python 3.9. I don't know what further we could do until/unless we figure out how to perform maintenance on eventlet; which is ongoing with no real timeline to when it'll get cleared up.
Thanks, Jay
On Fri, Dec 8, 2023 at 10:38 AM Jay Faulkner jay@gr-oss.io wrote:
There's no path for fixing that in eventlet upstream; which really means there's not a path for fixing this in OpenStack upstream. I suspect this means that we're just incompatible with python 3.6 as RHEL ships it unless RHEL also patches the eventlet shipped to fix compatibility.
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 9:09 AM Dmitry Tantsur dtantsur@protonmail.com wrote:
The thing is, eventlet is hacking into the Python internals. They are not guaranteed to never change. I did raise this problem to the Python team, and basically they consider fixing the CVE more important than keeping weird hacks working.
Dmitry
On 12/8/23 17:09, Jay Faulkner wrote:
Isn't this a bug in RedHat's distribution of Python? This seems like
the
kind of regression that would be guaranteed against in an LTS distribution. Can we get it addressed in that manner?
Thanks, Jay Faulkner
On Fri, Dec 8, 2023 at 6:10 AM Dmitry Tantsur <dtantsur@protonmail.com mailto:dtantsur@protonmail.com> wrote:
Hi stackers, I've noticed this some time ago, but somehow decided that itwouldn't
affect OpenStack upstream. Early sign of burn-out? Anyway... Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a waythat
breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486 <https://issues.redhat.com/browse/OCPBUGS-20486>). It manifests inthe
following traceback: Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py",line
173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File"/usr/lib/python3.6/site-packages/oslo_service/sslutils.py",
line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py",line
145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout' The fix should be rather simple IMO: just remove the condition on Python 2 from here:https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108 < https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108
. But I'm not sure, given the state of eventlet community, we can land anything these.
The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use anon-default
Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold... Thoughts, ideas? Dmitry
On 12/8/23 20:02, Jay Faulkner wrote:
I talked to Dmitry in IRC about this, patching the RPM in RHEL is done; but it's not helpful in this case as IPA ramdisks are built with pypi-copies of eventlet.
We talked about a potential solution of using python 3.9. I don't know what further we could do until/unless we figure out how to perform maintenance on eventlet; which is ongoing with no real timeline to when it'll get cleared up.
Thanks, Jay
Hi Jay,
FYI, Bookworm was released with Python 3.9, and the earliest version of IPA that I tried with it was 9.4.0. So at least, I can confirm that this version works with Python 3.9. Maybe earlier versions work too?
BTW, this is yet another point in favor of making the huge effort to get rid of the Eventlet ugly monkey-patching disaster... I really hope Itamar's plans will become reality.
I hope this helps, Cheers,
Thomas Goirand (zigo)
participants (3)
-
Dmitry Tantsur
-
Jay Faulkner
-
Thomas Goirand