[all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries
Hi all,
For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A... .
I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair
Hello,
This is what I’ve been concerned about regarding eventlet, when I was investigating adding a oslo.messaging driver for NATS that used asyncio [1] I got to dig down that rabbit hole and it would be an massive effort to do.
I haven’t thought about any other alternatives but asyncio is atleast a native Python feature which makes it a good fit, but we are talking duplicating all oslo code and modifying basically all code in the entire ecosystem.
It would be interesting to see if anybody has any alternative paths forward instead of that.
Best regards Tobias
[1] https://review.opendev.org/q/topic:asyncio-nats
On 9 Nov 2023, at 23:05, Jay Faulkner jay@gr-oss.io wrote:
Hi all,
For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A....
I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair
I've known for a while that eventlet had some issues, but I wasn't aware of just how much was or was not going on. I guess it all seemed to be moderately under control and acceptable until a bright light was shined on it. To me, Eventlet's state doesn't seem very acceptable for the long term health of OpenStack as a project, and all consumers of OpenStack and OpenStack component technologies. Thanks JayF for raising visibility and getting this discussion started.
I think regardless of path, we will likely need to take a phased approach, and see if there is any way we can plan out a "path".
Some things I think we should be thinking about:
* What is our short term path? What are the steps we can execute upon quickly without too much interruption to projects? * Is a fork to live for a few cycles acceptable to us, or not? * Are there intermediate steps we can take as we plan and begin executing on a long term plan? * Are there "smaller", more isolated, places we can start and move forward to a desired end state faster than others? * Are we okay with an intermediate end state? What is "good enough" in this case.
Obviously, there will be mixed opinions on this subject, and likely a need to use higher bandwidth discussion to reach any plan or consensus beyond getting initial thoughts and ideas into the open. I also think we can't "just wait" until the next PTG to reach consensus as a wider community.
Is anyone willing to champion the process of getting us to a plan? Asking because I know many of us in existing community leadership have very full, if not overflowing TODO lists and this is also a good opportunity to mentor and raise up a contributor or two who is not one of our existing community leaders.
-Julia
On Thu, Nov 9, 2023 at 3:52 PM Tobias Urdin tobias.urdin@binero.com wrote:
Hello,
This is what I’ve been concerned about regarding eventlet, when I was investigating adding a oslo.messaging driver for NATS that used asyncio [1] I got to dig down that rabbit hole and it would be an massive effort to do.
I haven’t thought about any other alternatives but asyncio is atleast a native Python feature which makes it a good fit, but we are talking duplicating all oslo code and modifying basically all code in the entire ecosystem.
It would be interesting to see if anybody has any alternative paths forward instead of that.
Best regards Tobias
[1] https://review.opendev.org/q/topic:asyncio-nats
On 9 Nov 2023, at 23:05, Jay Faulkner jay@gr-oss.io wrote:
Hi all,
For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A... .
I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair
On 11/10/23 15:31, Julia Kreger wrote:
I've known for a while that eventlet had some issues, but I wasn't aware of just how much was or was not going on. I guess it all seemed to be moderately under control and acceptable until a bright light was shined on it. To me, Eventlet's state doesn't seem very acceptable for the long term health of OpenStack as a project, and all consumers of OpenStack and OpenStack component technologies. Thanks JayF for raising visibility and getting this discussion started.
I think regardless of path, we will likely need to take a phased approach, and see if there is any way we can plan out a "path".
Some things I think we should be thinking about:
- What is our short term path? What are the steps we can execute upon
quickly without too much interruption to projects?
- Is a fork to live for a few cycles acceptable to us, or not?
- Are there intermediate steps we can take as we plan and begin
executing on a long term plan?
- Are there "smaller", more isolated, places we can start and move
forward to a desired end state faster than others?
- Are we okay with an intermediate end state? What is "good enough" in
this case.
Hi Julia,
For me, one obvious path is to get away from eventlet in the long run (thanks Sean, for suggesting gevent...). I'm fully aware that getting away from eventlet represent a lot of work. But at the same time, monkey patching, and accessing through private function of the Python standard lib is really horrible... and if there's *ONE* thing that I could with the project to work on, that'd be it. We've known this for years (since the beginning of OpenStack, in fact), and the technical dept is just accumulating. If this is kept unaddressed, it will continue to bite hard on each Python interpreter release.
But in the mean while, Python 3.12 support *MUST* be added to Eventlet ASAP. The transition to 3.12 in Debian Unstable (and soonish Ubuntu) has already started, and I'm getting bugs report related to Eventlet already (which I cannot fix until there's a build of greenlet with 3.12 support...).
Cheers,
Thomas Goirand (zigo)
On Wed, 2023-11-22 at 14:37 +0100, Thomas Goirand wrote:
On 11/10/23 15:31, Julia Kreger wrote:
I've known for a while that eventlet had some issues, but I wasn't aware of just how much was or was not going on. I guess it all seemed to be moderately under control and acceptable until a bright light was shined on it. To me, Eventlet's state doesn't seem very acceptable for the long term health of OpenStack as a project, and all consumers of OpenStack and OpenStack component technologies. Thanks JayF for raising visibility and getting this discussion started.
I think regardless of path, we will likely need to take a phased approach, and see if there is any way we can plan out a "path".
Some things I think we should be thinking about:
- What is our short term path? What are the steps we can execute upon
quickly without too much interruption to projects?
- Is a fork to live for a few cycles acceptable to us, or not?
- Are there intermediate steps we can take as we plan and begin
executing on a long term plan?
- Are there "smaller", more isolated, places we can start and move
forward to a desired end state faster than others?
- Are we okay with an intermediate end state? What is "good enough" in
this case.
Hi Julia,
For me, one obvious path is to get away from eventlet in the long run (thanks Sean, for suggesting gevent...). I'm fully aware that getting away from eventlet represent a lot of work. But at the same time, monkey patching, and accessing through private function of the Python standard lib is really horrible... and if there's *ONE* thing that I could with the project to work on, that'd be it. We've known this for years (since the beginning of OpenStack, in fact), and the technical dept is just accumulating. If this is kept unaddressed, it will continue to bite hard on each Python interpreter release.
well yes and no characterizing eventlet as technial debt when there has never be a agreement to stop using it or change our treading model is unfair. It is the selected technology today. openstack didn't start with eventlet we actually started with twisted and moved to eventlet for numerous reasons very early on. one of those was to avoid explicit concurrency in our code base. at the time it was consdier less error prone to use implicit concurrency and a framework that supported cooperative user land greenthreadign then explicit concurrency. im not going to comment on if that statement was correct but it was one fo the motivating factors.
if we as a community want to actully move away from eventlet to somethign else this is something we will need to do over the course of multiple release including support supporting both like across one slurp release.
This would have to be a desicsion driven across project and via a tc resolution or community goal like secure rbac.
all of the replacement other then gevnt requrie chaing the threading/concurrency model of all services that currently use eventlet form implicit concurrency to explicit concurrency https://docs.openstack.org/nova/latest/reference/threading.html
if we were to try and remove eventlet today one of the first thing we would need to do is extend oslo.service https://github.com/openstack/oslo.service so that it can be used without eventlet. in parallel to that we would have to convert any explict usage or eventlet in all the project and libs to either go via an absrction like futureist or use std lib fucnton or a diffenrt framework.
usiging stdlib modules is what i started doing in https://review.opendev.org/q/topic:%22gevnet%22 but its very non trivial and nova is one of the project that uses eventlet with out hevially depending on its internals
swift is very very tightly coupled to eventlet i belive and i suspect would be one of the hardest projects to rewirte into a non eventlet mode.
i generally would like to see us move to asyncio (possible with gevnet as a bridge) but it hard to justify investing the time to do that unless there is a agreement in the comunity that this will actully be done in all relevent projects and we can show a beifit to this to our users.
i would say having openstack actully work on 3.12/3.13 is a definitive benift to user but if you have a downstream product based on py39 it hard to justify to PMs why this must be done now and not in n years time when we are trying to build a product on 3.12. its short sighted but its a reality that rewrting a mature project like openstack to use a fundementally different execution model is unpalitable to those who want stablity. i can try to raise this internally again but unless its recognised at a TC/compunity level as a probelem that must be solved its hard to argue agaisn the status quo
But in the mean while, Python 3.12 support *MUST* be added to Eventlet ASAP. The transition to 3.12 in Debian Unstable (and soonish Ubuntu) has already started, and I'm getting bugs report related to Eventlet already (which I cannot fix until there's a build of greenlet with 3.12 support...).
Cheers,
Thomas Goirand (zigo)
* What is our short term path?
I think your short term path is to support and keep eventlet healthy in the short term. From there we have at least 4 options (ordered by order of preference that I think would benefit more to us): 1) getting involved in the current eventlet code base and trying fixing our py3.12 issues. That would be an occasion for some of us to gain the core maintainer rights. AFAIK some people have already proposed patches to solve our issues. The main problem with this option is the lack of active maintenance in the home repository of eventlet. This repository rests almost on the shoulder of one active maintainer and we should notice that nobody reviewed patches during almost 9 months. So IMO that option looks a bit hypothetical as it would also require current core maintainers contributions. The snake bites its tail. 2) We could engage in discussion with the current core maintainers to propose them to migrate eventlet into the openstack organization (maybe hosted into the oslo world), and then gain member rights to start making eventlet healthy again in the short term. If we decide to follow this path, I'd also suggest giving the maintainers rights to the current core members even if we move this project into our umbrella. This option would give us more flexibility to add new core members to this lib. The main issue with that option is that all the current PRs proposed to the current home repo would be lost if their authors do not volunteer to migrate their patches too... Tons of work would be lost and we would have to really well socialize the fact that the lib has been migrated to continue to benefit from the python community contributions and expertise. 3) if 1) 2) do not succeed, then we could decide to fork the eventlet lib into our organization (maybe hosted into the oslo world). I think that forking is the less desirable option as it would split the works and contributions between the eventlet existing repos. However, that won't be the first time that a project is forked and this option could lead us to keep eventlet healthy in the short term without too much effort either. We should also notice that all the contributions made to the original repos could be lost for us by forking this project. 4) Finally our last option is to see downstream distros maintainers fixing eventlet by patching it manually as already made in fedora ( https://src.fedoraproject.org/rpms/python-eventlet/blob/rawhide/f/python3.12...), however, we should notice that this option won't solve our upstream issue, as AFAIK, our requirements are pulled directly from pypi in the requested version. So that won't solve the issue in our gates, CI etc... Even if it is not really a solution I preferred to also refer to this option to list all our main paths.
As Julia previously noticed we are now close to the end of the year, so our activity will decrease during the next few weeks. That could be an opportunity to let time to see what's happening in the eventlet original repo and then decide early in 2024 between the previous options as well as the intent to investigate and put in place a better longer term solution.
* What are the steps we can execute upon quickly without too much interruption to projects?
I think the first step is to officialize and formalize this topic. We could start designing and propose a community goal to define short term and long term actions. We could design a decision tree with defined milestones to put in place our short term and long term solution and then prioritize things and get the big picture of the things to do move forward. This community goal can host the next discussions related to this topic as already made by example with the secure RBAC topic ( https://review.opendev.org/c/openstack/governance/+/799705).
I can initiate and manage this proposal and I volunteer to champion the process and to try to get us to a plan.
Also, depending on the outcome of the previous options, we could decide to delay the adoption of the python 3.12 runtimes. In other words, we could decide to ignore Python 3.12 during our next series (D). That will enlarge our maintenance windows and leave us more rooms to reach an appropriate solution.
* Is a fork to live for a few cycles acceptable to us, or not?
That would be one option but not the preferable one. IMO It would be preferable to either gain maintainers in the original repos, or migrate the official repos into the openstack org, but that decision mostly depends on the future activities and feedback of the current core maintainers of eventlet.
* Are there intermediate steps we can take as we plan and begin executing on a long term plan?
IMO, considering the gevent maintenance activity, the gevent solution could lead us in a near future to a similar situation that the current one faced with eventlet. I think we need to prioritize a more perennial solution and the stdlib asyncio one is a solution with battery included and more close to the python mantras and philosophy.
One first step could be to start forking oslo.service to migrate it to asyncio as a first step toward a long term plan.
However, we should notice that forking this kind of deliverables would increase the technical debt of the oslo world during a while, and likely it would necessitate patching both instances of the oslo deliverables if a bug or an issue is detected. AFAIK the activity on oslo.service has been a bit quiet since a while so I won't expect tons of issues suddenly, for this reason, I think this lib is a good starting point to reach an intermediate step.
Another point would be to define a standardized guide line to indicate to the community how to migrate their services. The Oslo migration could be a good opportunity for us to collect good practices and design the migration guide.
* Are there "smaller", more isolated, places we can start and move forward to a desired end state faster than others?
At first glance, I'd argue that the Oslo world could move faster than other places. Once the move on oslo would be done then, we could start migrating small services that are not relying too much to eventlet. Those services are not yet identified. I think we need the eyes of service experts, case by case, to identify who are the low hanging fruits first on the service side.
* Are we okay with an intermediate end state? What is "good enough" in this case.
An intermediate state could be: - the eventlet part is under control - oslo world is forked and migrated - a migration guide have been created based on our oslo observations - some services (easily migrated) have been fully transitioned to asyncio - at this point only the major services still relies on eventlet
As our main issue lives in mixing eventlet and asyncio, if a couple of services are fully migrated then we could start speaking of an accomplished intermediate state where the migration is well isolated and under control. The remaining services could then rely on our previous experiences and maybe their migration could be faster than initially expected.
To finish, as this topic seems to be sized to be delivered over multiple upstream releases, I'd imagine that py3.12 would be at some point supported by the coming versions of the downstream distros, so maybe we will be in sync with our intermediate state of migration that would make our PMs justifications easiers.
Thoughts?
As I wrote previously I'm volunteering to champion this topic and now I'm gonna start to initiate a (WIP) goal proposal.
Hi Herve,
Thanks for sharing your thoughts this way. It's a long message that probably took you a long time to write, I do appreciate the effort.
On 11/30/23 16:11, Herve Beraud wrote:
Finally our last option is to see downstream distros maintainers fixing eventlet by patching it manually
No, this isn't an option. We maintain already too many dirty patches for Eventlet, I don't have enough skills and time to understand all the dirty hacks of this ugly lib, etc.
I can take the Fedora patch and use it, for the moment, but that's not a solution in the long run.
As Julia previously noticed we are now close to the end of the year, so our activity will decrease during the next few weeks. That could be an opportunity to let time to see what's happening in the eventlet original repo and then decide early in 2024 between the previous options as well as the intent to investigate and put in place a better longer term solution.
That ship has sailed already... can we please move on to a more sustainable solution?
Also, depending on the outcome of the previous options, we could decide to delay the adoption of the python 3.12 runtimes.
This also mean completely ignoring downstream distros (including, but not limited to: Debian, Ubuntu...).
In other words, we could decide to ignore Python 3.12 during our next series (D).
D is probably what's going to end up in Trixie (aka: Debian 13). That's not an option for Debian at least, and probably not a viable option for Ubuntu either...
- Are there intermediate steps we can take as we plan and begin
executing on a long term plan?
IMO, considering the gevent maintenance activity, the gevent solution could lead us in a near future to a similar situation that the current one faced with eventlet. I think we need to prioritize a more perennial solution and the stdlib asyncio one is a solution with battery included and more close to the python mantras and philosophy.
+1
To finish, as this topic seems to be sized to be delivered over multiple upstream releases, I'd imagine that py3.12 would be at some point supported by the coming versions of the downstream distros
Debian is *already* transitioning to 3.12. I expect Ubuntu 24.04 to have 3.12 (only).
As I wrote previously I'm volunteering to champion this topic and now I'm gonna start to initiate a (WIP) goal proposal.
Thanks a lot. This kind of initiative is IMO much needed.
Cheers,
Thomas Goirand (zigo)
Hi,
As context, I've been working on submitting PRs to eventlet on behalf of G-Research OSS, with the initial goal of getting 3.12 support. Here's some thoughts on the problems I've seen with eventlet, and a potential solution, based on what I've learned:
## Short-term problem: lack of maintenance
Symptoms:
1. Tests don't pass, locally or in CI. 2. CI doesn't run at all for Python 3.11. 3. PR backlog.
If it's possible to add more maintainers, and put some engineering time into it, this is all solvable. In addition to any resources OpenStack companies to bring to bear there's also usually some potential maintainers among people who have submitted PRs. I've personally successfully revived one project (SparkMagic), initially just going through backlog, and then I managed to recruit someone who has been maintaining it for a few years.
It's possible that if a bunch of you opened an issue saying "hey we rely on eventlet, we understand you don't have time to work on this, can you give commit access to person X who we trust" you might get somewhere.
## Long-term problem: Monkeypatching is a technological dead-end
The premise of eventlet is drop-in compatibility via monkeypatching. Unfortunately that quite possibly hasn't been true for a long time, and it's becoming increasingly more difficult over time.
### Example #1: Compliance suite
Per the docs, "Eventlet provides the ability to test itself with the standard Python networking tests. This verifies that the libraries it wraps work at least as well as the standard ones do." (https://github.com/eventlet/eventlet/blob/master/doc/testing.rst#standard-li...)
That is, eventlet will run the Python standard library's test suite against eventlet to make sure it's compatible.
Unfortunately, this testing mechanism was never updated for Python 3.
As such, it's basically designed for Python 2.7, and there has been 13 major releases of Python since then. Is eventlet still compatible with the standard library? It's hard to say, but quite likely not.
### Example #2: RLock
When eventlet was originally written, `threading.RLock` was written in Python. This has a bug, e.g. it didn't actually work in the face of signals: https://bugs.python.org/issue13697 (there's a bunch of comments in the ticket from people encountering this in the real world, logging being a common situation.)
The problem doesn't occur in the version of RLock which is written in C, which is the current default and was introduced in Python 3.2.
However, the C version of RLock doesn't work with eventlet, so eventlet has been monkeypatching `threading.RLock`, replacing it with the (buggy and unfixable) version written in Python (`threading._PyRLock_`).
In 3.11 this gets worse, as the RLock version written in Python has become subtly incompatible with eventlet's expectations. To get the eventlet test suite passing on 3.11 I had to copy/paste the Python RLock code and tweak it (https://github.com/eventlet/eventlet/pull/823/files#diff-029df1ae9b7431e9cdd...).
So now eventlet has to use a forked version of a broken implementation of RLock. It's possible there's another solution, but eventlet basically relies on monkeypatching a whole bunch of functions and on implementation details of Python standard library using those functions in particular ways, which are not always stable over time.
This problem will continue to get worse as Python evolves. E.g. it would not surprise if the GIL removal makes things even more difficult for eventlet.
Is all this solvable? Yes, but at at a potentially significant engineering cost.
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops, in this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
1. Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event loop. 2. Create a little bit of glue so that eventlet code can wait for the result of a coroutine object (i.e. the result of calling an `async def` function). 3. Make `eventlet.greenlet.GreenThread` something you can `await` in an `async def` function.
The end result is that asyncio code can call eventlet code, and eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a gradual manner both within and across projects:
1. Within an OpenStack project, code could be gradually switched to async functions and asyncio libraries. 2. When an OpenStack library fully migrates to asyncio, it will still be usable by anything that is still running on eventlet.
On Thu, 2023-11-30 at 17:11 +0000, itamar@pythonspeed.com wrote:
Hi,
As context, I've been working on submitting PRs to eventlet on behalf of G-Research OSS, with the initial goal of getting 3.12 support. Here's some thoughts on the problems I've seen with eventlet, and a potential solution, based on what I've learned:
## Short-term problem: lack of maintenance
Symptoms:
- Tests don't pass, locally or in CI.
- CI doesn't run at all for Python 3.11.
- PR backlog.
If it's possible to add more maintainers, and put some engineering time into it, this is all solvable. In addition to any resources OpenStack companies to bring to bear there's also usually some potential maintainers among people who have submitted PRs. I've personally successfully revived one project (SparkMagic), initially just going through backlog, and then I managed to recruit someone who has been maintaining it for a few years.
It's possible that if a bunch of you opened an issue saying "hey we rely on eventlet, we understand you don't have time to work on this, can you give commit access to person X who we trust" you might get somewhere.
## Long-term problem: Monkeypatching is a technological dead-end
The premise of eventlet is drop-in compatibility via monkeypatching. Unfortunately that quite possibly hasn't been true for a long time, and it's becoming increasingly more difficult over time.
### Example #1: Compliance suite
Per the docs, "Eventlet provides the ability to test itself with the standard Python networking tests. This verifies that the libraries it wraps work at least as well as the standard ones do." (https://github.com/eventlet/eventlet/blob/master/doc/testing.rst#standard-li...)
That is, eventlet will run the Python standard library's test suite against eventlet to make sure it's compatible.
Unfortunately, this testing mechanism was never updated for Python 3.
As such, it's basically designed for Python 2.7, and there has been 13 major releases of Python since then. Is eventlet still compatible with the standard library? It's hard to say, but quite likely not.
### Example #2: RLock
When eventlet was originally written, `threading.RLock` was written in Python. This has a bug, e.g. it didn't actually work in the face of signals: https://bugs.python.org/issue13697%C2%A0(there%27s a bunch of comments in the ticket from people encountering this in the real world, logging being a common situation.)
The problem doesn't occur in the version of RLock which is written in C, which is the current default and was introduced in Python 3.2.
However, the C version of RLock doesn't work with eventlet, so eventlet has been monkeypatching `threading.RLock`, replacing it with the (buggy and unfixable) version written in Python (`threading._PyRLock_`).
In 3.11 this gets worse, as the RLock version written in Python has become subtly incompatible with eventlet's expectations. To get the eventlet test suite passing on 3.11 I had to copy/paste the Python RLock code and tweak it (https://github.com/eventlet/eventlet/pull/823/files#diff-029df1ae9b7431e9cdd... 8816R427).
So now eventlet has to use a forked version of a broken implementation of RLock. It's possible there's another solution, but eventlet basically relies on monkeypatching a whole bunch of functions and on implementation details of Python standard library using those functions in particular ways, which are not always stable over time.
This problem will continue to get worse as Python evolves. E.g. it would not surprise if the GIL removal makes things even more difficult for eventlet.
Is all this solvable? Yes, but at at a potentially significant engineering cost.
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops, in this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio.
At this point asyncio and eventlet code can run in the same event loop. 2. Create a little bit of glue so that eventlet code can wait for the result of a coroutine object (i.e. the result of calling an `async def` function). 3. Make `eventlet.greenlet.GreenThread` something you can `await` in an `async def` function.
The end result is that asyncio code can call eventlet code, and eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still be usable by anything that is still running on
eventlet.
so long term i think that is the approch we should take and i agree that mediaum ot short term the base way to do that is likely to create oslo.aiohub or similar that has an asyncio backed eventloop that we can enable instead fo the default eventlet one. realsitcally howeer we really do need to drive this as cross community effort and we need to get peopel form multiple teams to sign up to do that work. im not really sure i can get invovled in this in cycle but if there were patches to nova proposed to optionally enabel this alternitive hub i would review them and i thinkt his is something we could set a s a comunity goal to try and do in D/E i know that is late for distros but 3.12 is not in the set of python runtimes for caracal https://governance.openstack.org/tc/reference/runtimes/2024.1.html i woudl like to see it in the required runtimes for D 2024.2 and if we want that to happen we need to resolve the enventlet issues.
starting in 2024.1 to enable as much 3.12 support as we can is not a bad thing im just not sure its something upstream can really commit to doing before the D/E cycle.
itamar@pythonspeed.com writes:
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops, in this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event loop.
- Create a little bit of glue so that eventlet code can wait for the result of a coroutine object (i.e. the result of calling an `async def` function).
- Make `eventlet.greenlet.GreenThread` something you can `await` in an `async def` function.
The end result is that asyncio code can call eventlet code, and eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still be usable by anything that is still running on eventlet.
Thanks for taking the time to write this part out in detail. I had an off-list conversation with Herve recently and I was starting to flesh out something quite similar to what you've proposed here. I'm not terribly familiar (yet) with all of the finer details surrounding this so this really helps to point me in the right direction and is motivating that you are thinking along the same lines.
I think it's going to be 100% necessary to run both asyncio and eventlet code simultaneously within OpenStack services for some time. Trying to maintain both in parallel and then hard-cut over to asyncio at some point sounds like an absolute nightmare and effectively impossible to pull off. We'll have to do it bit by bit over a rather extended timeframe, and even then I suspect there is untold pain ahead.
eck
Hello,
Thanks Itamar for your previous email and thanks for the useful information.
FYI I just submitted a first draft (still WIP) of the related community goal proposal, it can be found here https://review.opendev.org/c/openstack/governance/+/902585.
Thanks also Sean and John for the discussion about the aiohub. I included this part of the solution into the proposal. However I think we need more details about your vision. That would help to draft the specs of this solution. I think we could start to centralize this topic somewhere, so, I created a dedicated section into the wiki page I created to track the progress of this community goal. As you seem interested in this topic, do you mind giving us more details and thoughts about the aiohub and about how you imagine its design and its implementation. Do not hesitate to fullfile it with your observations and your discoveries. I think this kind of discussion can be run in parallel with the global goal proposal.
I'll continue to work on the proposal in the coming days, and I'll remove the WIP flag once I think it has reached a good state, then we will be able to follow the normal process and the acceptance, or not, of this proposal.
Thanks for your time.
Le jeu. 30 nov. 2023 à 20:38, John Eckersberg jeckersb@redhat.com a écrit :
itamar@pythonspeed.com writes:
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops, in
this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event
loop.
- Create a little bit of glue so that eventlet code can wait for the
result of a coroutine object (i.e. the result of calling an `async def` function).
- Make `eventlet.greenlet.GreenThread` something you can `await` in an
`async def` function.
The end result is that asyncio code can call eventlet code, and eventlet
code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a
gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to
async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still be
usable by anything that is still running on eventlet.
Thanks for taking the time to write this part out in detail. I had an off-list conversation with Herve recently and I was starting to flesh out something quite similar to what you've proposed here. I'm not terribly familiar (yet) with all of the finer details surrounding this so this really helps to point me in the right direction and is motivating that you are thinking along the same lines.
I think it's going to be 100% necessary to run both asyncio and eventlet code simultaneously within OpenStack services for some time. Trying to maintain both in parallel and then hard-cut over to asyncio at some point sounds like an absolute nightmare and effectively impossible to pull off. We'll have to do it bit by bit over a rather extended timeframe, and even then I suspect there is untold pain ahead.
eck
Giving the list a quick update.
The Eventlet maintainer has given access to multiple members of the eventlet community, including several OpenStack-associated contributors (including Herve and Itamar) and several existing eventlet contributors who did not already have maintainer access.
It looks like we're going to have a path forward with getting the eventlet library maintained again, and working towards building in a migration path.
Thanks to Temoto (the longtime, decade+ eventlet developer), Herve and others who have taken this issue seriously and helped to ensure things will keep moving forward for a while longer.
- Jay Faulkner OpenStack TC Chair
On Mon, Dec 4, 2023 at 10:49 AM Herve Beraud hberaud@redhat.com wrote:
Hello,
Thanks Itamar for your previous email and thanks for the useful information.
FYI I just submitted a first draft (still WIP) of the related community goal proposal, it can be found here https://review.opendev.org/c/openstack/governance/+/902585.
Thanks also Sean and John for the discussion about the aiohub. I included this part of the solution into the proposal. However I think we need more details about your vision. That would help to draft the specs of this solution. I think we could start to centralize this topic somewhere, so, I created a dedicated section into the wiki page I created to track the progress of this community goal. As you seem interested in this topic, do you mind giving us more details and thoughts about the aiohub and about how you imagine its design and its implementation. Do not hesitate to fullfile it with your observations and your discoveries. I think this kind of discussion can be run in parallel with the global goal proposal.
I'll continue to work on the proposal in the coming days, and I'll remove the WIP flag once I think it has reached a good state, then we will be able to follow the normal process and the acceptance, or not, of this proposal.
Thanks for your time.
Le jeu. 30 nov. 2023 à 20:38, John Eckersberg jeckersb@redhat.com a écrit :
itamar@pythonspeed.com writes:
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops, in
this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event
loop.
- Create a little bit of glue so that eventlet code can wait for the
result of a coroutine object (i.e. the result of calling an `async def` function).
- Make `eventlet.greenlet.GreenThread` something you can `await` in an
`async def` function.
The end result is that asyncio code can call eventlet code, and
eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a
gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to
async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still
be usable by anything that is still running on eventlet.
Thanks for taking the time to write this part out in detail. I had an off-list conversation with Herve recently and I was starting to flesh out something quite similar to what you've proposed here. I'm not terribly familiar (yet) with all of the finer details surrounding this so this really helps to point me in the right direction and is motivating that you are thinking along the same lines.
I think it's going to be 100% necessary to run both asyncio and eventlet code simultaneously within OpenStack services for some time. Trying to maintain both in parallel and then hard-cut over to asyncio at some point sounds like an absolute nightmare and effectively impossible to pull off. We'll have to do it bit by bit over a rather extended timeframe, and even then I suspect there is untold pain ahead.
eck
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
Hello,
I'm happy to announce that we just created a new release of eventlet (0.34.1):
- https://pypi.org/project/eventlet/0.34.1/ - https://github.com/eventlet/eventlet/issues/842
The main achievement made in this new release is the fix of the compatibility for Python 3.12.
Here are all the things released within this new version:
0.34.1 =====
* [bug] Fix memory leak in greendns https://github.com/eventlet/eventlet/issues/810 * [infra] Fix OIDC authentication failure https://github.com/eventlet/eventlet/pull/855 * [bug] Ignore asyncore and asynchat for Python 3.12+ https://github.com/eventlet/eventlet/issues/804
0.34.0 (Not released on Pypi) ======================
* Dropped support for Python 3.6 and earlier. * Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter' https://github.com/eventlet/eventlet/pull/847 * Add support of Python 3.12 https://github.com/eventlet/eventlet/pull/817 * Drop unmaintained and unused stdlib tests https://github.com/eventlet/eventlet/pull/820 * Fix tests and CI for Python 3.7 and higher https://github.com/eventlet/eventlet/pull/831 and https://github.com/eventlet/eventlet/pull/832 * Stop claiming to create universal wheels https://github.com/eventlet/eventlet/pull/841 * Fix green logging locks for Python versions <= 3.10 https://github.com/eventlet/eventlet/pull/754
Openstack requirements are not yet updated, but I just proposed a patch to update the version of eventlet in Openstack: https://review.opendev.org/c/openstack/requirements/+/904147
Hopefully it will solve all the problems described in this thread.
Cheers
Le mer. 13 déc. 2023 à 20:26, Jay Faulkner jay@gr-oss.io a écrit :
Giving the list a quick update.
The Eventlet maintainer has given access to multiple members of the eventlet community, including several OpenStack-associated contributors (including Herve and Itamar) and several existing eventlet contributors who did not already have maintainer access.
It looks like we're going to have a path forward with getting the eventlet library maintained again, and working towards building in a migration path.
Thanks to Temoto (the longtime, decade+ eventlet developer), Herve and others who have taken this issue seriously and helped to ensure things will keep moving forward for a while longer.
Jay Faulkner OpenStack TC Chair
On Mon, Dec 4, 2023 at 10:49 AM Herve Beraud hberaud@redhat.com wrote:
Hello,
Thanks Itamar for your previous email and thanks for the useful information.
FYI I just submitted a first draft (still WIP) of the related community goal proposal, it can be found here https://review.opendev.org/c/openstack/governance/+/902585.
Thanks also Sean and John for the discussion about the aiohub. I included this part of the solution into the proposal. However I think we need more details about your vision. That would help to draft the specs of this solution. I think we could start to centralize this topic somewhere, so, I created a dedicated section into the wiki page I created to track the progress of this community goal. As you seem interested in this topic, do you mind giving us more details and thoughts about the aiohub and about how you imagine its design and its implementation. Do not hesitate to fullfile it with your observations and your discoveries. I think this kind of discussion can be run in parallel with the global goal proposal.
I'll continue to work on the proposal in the coming days, and I'll remove the WIP flag once I think it has reached a good state, then we will be able to follow the normal process and the acceptance, or not, of this proposal.
Thanks for your time.
Le jeu. 30 nov. 2023 à 20:38, John Eckersberg jeckersb@redhat.com a écrit :
itamar@pythonspeed.com writes:
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops,
in this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event
loop.
- Create a little bit of glue so that eventlet code can wait for the
result of a coroutine object (i.e. the result of calling an `async def` function).
- Make `eventlet.greenlet.GreenThread` something you can `await` in
an `async def` function.
The end result is that asyncio code can call eventlet code, and
eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a
gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to
async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still
be usable by anything that is still running on eventlet.
Thanks for taking the time to write this part out in detail. I had an off-list conversation with Herve recently and I was starting to flesh out something quite similar to what you've proposed here. I'm not terribly familiar (yet) with all of the finer details surrounding this so this really helps to point me in the right direction and is motivating that you are thinking along the same lines.
I think it's going to be 100% necessary to run both asyncio and eventlet code simultaneously within OpenStack services for some time. Trying to maintain both in parallel and then hard-cut over to asyncio at some point sounds like an absolute nightmare and effectively impossible to pull off. We'll have to do it bit by bit over a rather extended timeframe, and even then I suspect there is untold pain ahead.
eck
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
To be explicit: https://review.opendev.org/c/openstack/governance/+/902585 still exists, and we are still making plans to migrate off eventlet in the medium term. Thanks to Herve, Itamar, and the existing eventlet community for stepping up and resetting the timer on this migration!
Thanks, Jay Faulkner OpenStack TC Chair
On Wed, Dec 20, 2023 at 11:08 AM Herve Beraud hberaud@redhat.com wrote:
Hello,
I'm happy to announce that we just created a new release of eventlet (0.34.1):
The main achievement made in this new release is the fix of the compatibility for Python 3.12.
Here are all the things released within this new version:
0.34.1
- [bug] Fix memory leak in greendns
https://github.com/eventlet/eventlet/issues/810
- [infra] Fix OIDC authentication failure
https://github.com/eventlet/eventlet/pull/855
- [bug] Ignore asyncore and asynchat for Python 3.12+
https://github.com/eventlet/eventlet/issues/804
0.34.0 (Not released on Pypi)
- Dropped support for Python 3.6 and earlier.
- Fix Python 3.13 compat by adding missing attibute '_is_main_interpreter'
https://github.com/eventlet/eventlet/pull/847
- Add support of Python 3.12 https://github.com/eventlet/eventlet/pull/817
- Drop unmaintained and unused stdlib tests
https://github.com/eventlet/eventlet/pull/820
- Fix tests and CI for Python 3.7 and higher
https://github.com/eventlet/eventlet/pull/831 and https://github.com/eventlet/eventlet/pull/832
- Stop claiming to create universal wheels
https://github.com/eventlet/eventlet/pull/841
- Fix green logging locks for Python versions <= 3.10
https://github.com/eventlet/eventlet/pull/754
Openstack requirements are not yet updated, but I just proposed a patch to update the version of eventlet in Openstack: https://review.opendev.org/c/openstack/requirements/+/904147
Hopefully it will solve all the problems described in this thread.
Cheers
Le mer. 13 déc. 2023 à 20:26, Jay Faulkner jay@gr-oss.io a écrit :
Giving the list a quick update.
The Eventlet maintainer has given access to multiple members of the eventlet community, including several OpenStack-associated contributors (including Herve and Itamar) and several existing eventlet contributors who did not already have maintainer access.
It looks like we're going to have a path forward with getting the eventlet library maintained again, and working towards building in a migration path.
Thanks to Temoto (the longtime, decade+ eventlet developer), Herve and others who have taken this issue seriously and helped to ensure things will keep moving forward for a while longer.
Jay Faulkner OpenStack TC Chair
On Mon, Dec 4, 2023 at 10:49 AM Herve Beraud hberaud@redhat.com wrote:
Hello,
Thanks Itamar for your previous email and thanks for the useful information.
FYI I just submitted a first draft (still WIP) of the related community goal proposal, it can be found here https://review.opendev.org/c/openstack/governance/+/902585.
Thanks also Sean and John for the discussion about the aiohub. I included this part of the solution into the proposal. However I think we need more details about your vision. That would help to draft the specs of this solution. I think we could start to centralize this topic somewhere, so, I created a dedicated section into the wiki page I created to track the progress of this community goal. As you seem interested in this topic, do you mind giving us more details and thoughts about the aiohub and about how you imagine its design and its implementation. Do not hesitate to fullfile it with your observations and your discoveries. I think this kind of discussion can be run in parallel with the global goal proposal.
I'll continue to work on the proposal in the coming days, and I'll remove the WIP flag once I think it has reached a good state, then we will be able to follow the normal process and the acceptance, or not, of this proposal.
Thanks for your time.
Le jeu. 30 nov. 2023 à 20:38, John Eckersberg jeckersb@redhat.com a écrit :
itamar@pythonspeed.com writes:
## A potential path for migrating off of eventlet
Like many networking frameworks, eventlet has pluggable event loops,
in this case called a "hub". Typically these wrap things like select() and epoll(), but there also used to be a hub that ran on Twisted.
My hypothesis is that it should be possible to:
- Create a hub that runs on asyncio. At this point asyncio and eventlet code can run in the same event
loop.
- Create a little bit of glue so that eventlet code can wait for the
result of a coroutine object (i.e. the result of calling an `async def` function).
- Make `eventlet.greenlet.GreenThread` something you can `await` in
an `async def` function.
The end result is that asyncio code can call eventlet code, and
eventlet code can call asyncio code. There is an existing integration on this model for gevent, as reference: https://pypi.org/project/asyncio-gevent/
If this works, it would allow migrating from eventlet to asyncio in a
gradual manner both within and across projects:
- Within an OpenStack project, code could be gradually switched to
async functions and asyncio libraries.
- When an OpenStack library fully migrates to asyncio, it will still
be usable by anything that is still running on eventlet.
Thanks for taking the time to write this part out in detail. I had an off-list conversation with Herve recently and I was starting to flesh out something quite similar to what you've proposed here. I'm not terribly familiar (yet) with all of the finer details surrounding this so this really helps to point me in the right direction and is motivating that you are thinking along the same lines.
I think it's going to be 100% necessary to run both asyncio and eventlet code simultaneously within OpenStack services for some time. Trying to maintain both in parallel and then hard-cut over to asyncio at some point sounds like an absolute nightmare and effectively impossible to pull off. We'll have to do it bit by bit over a rather extended timeframe, and even then I suspect there is untold pain ahead.
eck
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
On Thu, 2023-11-09 at 14:05 -0800, Jay Faulkner wrote:
Hi all,
For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817%C2%A0and a PR up here: https://github.com/eventlet/eventlet/pull/820%C2%A0to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638%C2%A0there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A... .
I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem.
actully i have one proposal at lesat for the medium to short term and that is to enabel supprot for gevent i did some expeiemtns back in september when i was on vacation with the aim of seeing what would be required to evenutlaly replacing eventlet with genvet in nova https://review.opendev.org/q/topic:gevnet the idea was that we could use https://pypi.org/project/asyncio-gevent/ enable use to eventually move to asincio via gevent as a bridge.
gevent 23.7.0 added intiall support for py3.12 which was approved impoved in 23.9.1
my understanding is gevent has less technial debth, its build on libuv which os the same eventoloop that power node.js https://codeahoy.com/learn/libuv/ch1/ is closer to the stand lib in terems or api and i think generally less problematic to port to newer python versoions as a result.
asyncio-gevent woudl also enabel those interested in adding nats support to oslo messaging to actully do that as it solve the problem that we cant mix asincio and eventlet.
asyncio-gevent isnt super well mainteined unfortunetly but its also a rather small shim required to make both models work togehter and entirly optional if we just wanted to mvoe from eventlet to gevent.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair
I realize this is a "holiday week" for most folks in North America, but I wonder if we should schedule a meeting to begin to build some consensus on a forward path?
It is also worth noting we are quickly approaching the end of the year, and all of the various challenges with scheduling which tend to happen during this time of year.
-Julia
On Fri, Nov 10, 2023 at 10:12 AM smooney@redhat.com wrote:
On Thu, 2023-11-09 at 14:05 -0800, Jay Faulkner wrote:
Hi all,
For those who aren't aware, eventlet (
https://github.com/eventlet/eventlet)
is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the
#openstack-tc
IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12
support
for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right
now.
There have been no commits to the primary branch in nearly eight months
and
it's been a single maintainer project for an extremely long time. There
are
also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing
unit
tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic
database
migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed:
https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A...
.
I don't have a specific proposal for a solution. I do know that right
now,
OpenStack services are built "all-in" on eventlet, with no obvious or
easy
migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this
to raise an alarm about this situation, and to hope that community
members
can work together to solve the problem.
actully i have one proposal at lesat for the medium to short term and that is to enabel supprot for gevent i did some expeiemtns back in september when i was on vacation with the aim of seeing what would be required to evenutlaly replacing eventlet with genvet in nova https://review.opendev.org/q/topic:gevnet the idea was that we could use https://pypi.org/project/asyncio-gevent/ enable use to eventually move to asincio via gevent as a bridge.
gevent 23.7.0 added intiall support for py3.12 which was approved impoved in 23.9.1
my understanding is gevent has less technial debth, its build on libuv which os the same eventoloop that power node.js https://codeahoy.com/learn/libuv/ch1/ is closer to the stand lib in terems or api and i think generally less problematic to port to newer python versoions as a result.
asyncio-gevent woudl also enabel those interested in adding nats support to oslo messaging to actully do that as it solve the problem that we cant mix asincio and eventlet.
asyncio-gevent isnt super well mainteined unfortunetly but its also a rather small shim required to make both models work togehter and entirly optional if we just wanted to mvoe from eventlet to gevent.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair
participants (8)
-
Herve Beraud
-
itamar＠pythonspeed.com
-
Jay Faulkner
-
John Eckersberg
-
Julia Kreger
-
smooney＠redhat.com
-
Thomas Goirand
-
Tobias Urdin