I've known for a while that eventlet had some issues, but I wasn't aware of just how much was or was not going on. I guess it all seemed to be moderately under control and acceptable until a bright light was shined on it. To me, Eventlet's state doesn't seem very acceptable for the long term health of OpenStack as a project, and all consumers of OpenStack and OpenStack component technologies. Thanks JayF for raising visibility and getting this discussion started. I think regardless of path, we will likely need to take a phased approach, and see if there is any way we can plan out a "path". Some things I think we should be thinking about: * What is our short term path? What are the steps we can execute upon quickly without too much interruption to projects? * Is a fork to live for a few cycles acceptable to us, or not? * Are there intermediate steps we can take as we plan and begin executing on a long term plan? * Are there "smaller", more isolated, places we can start and move forward to a desired end state faster than others? * Are we okay with an intermediate end state? What is "good enough" in this case. Obviously, there will be mixed opinions on this subject, and likely a need to use higher bandwidth discussion to reach any plan or consensus beyond getting initial thoughts and ideas into the open. I also think we can't "just wait" until the next PTG to reach consensus as a wider community. Is anyone willing to champion the process of getting us to a plan? Asking because I know many of us in existing community leadership have very full, if not overflowing TODO lists and this is also a good opportunity to mentor and raise up a contributor or two who is not one of our existing community leaders. -Julia On Thu, Nov 9, 2023 at 3:52 PM Tobias Urdin <tobias.urdin@binero.com> wrote:
Hello,
This is what I’ve been concerned about regarding eventlet, when I was investigating adding a oslo.messaging driver for NATS that used asyncio [1] I got to dig down that rabbit hole and it would be an massive effort to do.
I haven’t thought about any other alternatives but asyncio is atleast a native Python feature which makes it a good fit, but we are talking duplicating all oslo code and modifying basically all code in the entire ecosystem.
It would be interesting to see if anybody has any alternative paths forward instead of that.
Best regards Tobias
[1] https://review.opendev.org/q/topic:asyncio-nats
On 9 Nov 2023, at 23:05, Jay Faulkner <jay@gr-oss.io> wrote:
Hi all,
For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading.
A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today.
First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12.
Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it.
Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3A... .
I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem.
Thanks for reading and your help, Jay Faulkner OpenStack TC Chair