Search results for "#eventlet-removal" - openstack-discuss

Re: #eventlet-removal - Progress & Update

by Herve Beraud

Le lun. 14 oct. 2024 à 13:05, Dmitry Tantsur <dtantsur(a)protonmail.com> a écrit : > Hi, > > On 10/9/24 9:36 AM, Herve Beraud wrote: > > Hey there, > > > > @Arnaud: Mistral is already in our list of identified deliverables: > > https://wiki.openstack.org/wiki/Eventlet- > > removal#Map_of_the_Impacted_Deliverables <https://wiki.openstack.org/ > > wiki/Eventlet-removal#Map_of_the_Impacted_Deliverables>. But thanks for > > the heads up. > > By the way, was this list compiled based on a text search? It lists a > few *-specs repositories which are unlikely to be impacted. The mentions > of puppet are also suspicious. > Indeed, this list was created by using beagle ( https://beagle-hound.readthedocs.io/en/latest/user/index.html) so it includes all kinds of occurence, even specs repository. I think we can ignore this kind of occurence, but I think they can also be useful to help us to identify some previous architectural decisions. To reduce the noise, I'll do another round of filtering to keep only legit occurrences, and I'll put the specs "results" into a separated section. Do not hesitate to use Beagle for our deliverables. Beagle allows you to ignore files, and allows you to use matching patterns. > From the ironic side, I suspect we won't be migrating ironic-inspector > and its client as they are deprecated. I haven't discussed it with the > team though. > > Dmitry > > > > > @engineer2024: I'll let the Nova team answer your question. > > > > Le mar. 8 oct. 2024 à 22:56, Arnaud Morin <arnaud.morin(a)gmail.com > > <mailto:arnaud.morin@gmail.com>> a écrit : > > > > Hey Herve, > > > > Please add mistral to your list. > > I havn't dig into code for this yet, but it's in requirements and it > > seems to be used by some of the services. > > > > Regards, > > Arnaud. > > > > On 04.10.24 - 17:08, Herve Beraud wrote: > > > We’re excited to share the latest updates on our ongoing efforts > > to phase > > > out Eventlet across our various OpenStack projects. Below you’ll > > find a > > > summary of our progress, key milestones, and important dates to > > keep in > > > mind as we move forward with this critical initiative. > > > > > > ## Current Status > > > > > > We’ve completed the initial analysis phase across more than 120 > > OpenStack > > > projects. This includes identifying the various patterns where > > Eventlet is > > > currently in use and evaluating where it can be safely removed or > > disabled. > > > Here are some highlights: > > > > > > - Completed Analysis: Projects where Eventlet is used include > > (but are not > > > limited to): Aodh, Barbican, Nova, Neutron, Tacker, Taskflow, > > Trove, Venus, > > > Watcher, and Zun. > > > > > > - Eventlet Optional: Several projects have been identified where > > Eventlet > > > seems to be optional and can be disabled with minimal impact. For > > these > > > projects, we will move forward the discussion with teams to try > > > deactivating Eventlet in upcoming updates. Once done, projects of > > this kind > > > would be able to start their migration with more serenity. > > > > > > - Eventlet Required: Some projects still rely heavily on Eventlet > > for core > > > functionality. For these, we are going to develop specific > > migration plans. > > > > > > ## Migration Plans in Progress > > > > > > - oslo.db: Preparing its Asyncio based engine facade; > > > - oslo.service: Refactoring plans being discussed to minimize > > disruption > > > during the migration; > > > - glance: Refactoring code to use native thread model rather than > > Eventlet > > > > > > More details can be found here: > > > https://review.opendev.org/q/topic:%22eventlet-removal%22 > > <https://review.opendev.org/q/topic:%22eventlet-removal%22> > > > > > > ## Latest Updates > > > > > > - Community Goal Transition: The ongoing community goal for > removing > > > Eventlet is now in the process of becoming an official "selected" > > goal, > > > https://review.opendev.org/c/openstack/governance/+/931254 > > <https://review.opendev.org/c/openstack/governance/+/931254>. > > > > > > - Pattern Identification Completed: We’ve identified key patterns > > related > > > to Eventlet usage, including its common integration with > > monkey_patch(), > > > eventlet.Timeout, eventlet.GreenPool, and eventlet.wsgi.server(). > > Visit the > > > official wiki to get more details > > > https://wiki.openstack.org/wiki/Eventlet- > > removal#Identifying_Common_Pattern <https://wiki.openstack.org/wiki/ > > Eventlet-removal#Identifying_Common_Pattern> > > > > > > ## Important Dates > > > > > > - October 21st: cross PTG session. > > > - October 23rd: cross PTG session (Q&A session and Open > > Discussion session) > > > > > > ## Call to Action > > > > > > We need your involvement to accelerate the progress of this > > initiative! If > > > your project is affected by Eventlet, now is the time to start > > planning > > > your migration. Teams are encouraged to take immediate steps in > > assessing > > > how to deactivate Eventlet or to transition to alternative > solutions. > > > > > > We defined key roles and responsibilities. Those will help us to > > lower the > > > cost of the migration for our community. Teams are invited to > > assign them > > > now. Being ready as soon as possible will allow you to better take > > > advantage of the coming PTG sessions. > > > > > > Please find more details about these roles and their short terms > > tasks: > > > https://wiki.openstack.org/wiki/Eventlet- > > removal#Roles_and_Responsibilities <https://wiki.openstack.org/wiki/ > > Eventlet-removal#Roles_and_Responsibilities> > > > > > > ## Final Thoughts > > > > > > This is a significant step forward for improving the stability, > > > scalability, and maintainability of our projects. With your > continued > > > support, we’re confident that this migration will bring > substantial > > > benefits to the OpenStack community. > > > > > > If there are any questions or if you need assistance with your > > project’s > > > transition away from Eventlet, please join the discussion on the > > > #eventlet-removal channels (OpenStack irc, RedHat slack, mailing > > list). > > > > > > Thank you for your dedication and hard work! > > > > > > -- > > > Hervé Beraud > > > Senior Software Engineer at Red Hat > > > irc: hberaud > > > https://github.com/4383/ <https://github.com/4383/> > > > > > > > > -- > > Hervé Beraud > > Senior Software Engineer at Red Hat > > irc: hberaud > > https://github.com/4383/ <https://github.com/4383/> > > > > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

9 months, 4 weeks

Re: #eventle-removal - Progress & Updates Epoxy Milestones 2-3

by Sean Mooney

On 24/02/2025 13:45, Herve Beraud wrote: > Hey Walter, > > Thank you for your initiative, that's the right way to go. > I added myself into the reviewers of your patch. > Concerning timeout usages, here is an example of one option Mistral > used to drop them https://review.opendev.org/c/openstack/mistral/+/942058 > Let us know if you want us to setup a deeper discussion about tiimeout > during our next meeting > https://wiki.openstack.org/wiki/Eventlet-removal#Recurring_Events > > Le mar. 18 févr. 2025 à 15:58, Walter Boring <waboring(a)hemna.com> a > écrit : > > Hello there, > I have added a WIP patch for oslo.vmware in gerrit to remove > oslo.vmware's reliance on eventlet. Both cinder and nova use > oslo.vmware to support vmware as a valid backend. > I would not consider oslo.vmware to be experimental as it's been > around for ages. The only thing I might need help with is the > replacement of eventlet.Timeout usage. > Looks like nova still relies on eventlet.timeout as well. > Nova will not be deprecating the use of eventlet until 2026.1 nova is not capable of rungin with out eventlet in 2025.1 and we will not do the deprecation until that is possible. we may or may not be able to run without eventlet in 2025.2 but our initall target to support using the non eventlet backend of oslo.servce ectra will be 2026.1 the earlist we might remove supprot for eventlet is 2026.2 currently we do not have poeple working on this. looping call is part of the top level oslo.service api and will be ported to using the impelmation from futurist in the non eventlet backend so as long oslo.vmware is usign the oslo.servie veriosn that should be transparent. https://github.com/openstack/oslo.service/blob/master/oslo_service/loopingc… the eventlet.Timeout usage however ves is harder to replace. i think it only used here https://github.com/openstack/oslo.vmware/blob/face67836e6d9c7eb63cb52e8120b… that likely can be delegated to a background io thread pool and a future can be return which can be waited on. ideally the time out would be proagated to the underlying soap/rest client that is doign the upload so the connection can be closed if the timeout is exceeded butim not famialr with exactly how that works in this libary. in general lib like oslo.vmware should avoid using eventlet or looping call directly but oslo.vmware has a few release to transition to the new model. this does nto need to be doen before Epoxy milestone 3 > > > Not sure if my WIP is the right way to go, but since oslo.service > has an abstraction already in place to replace eventlet, my WIP > relies on that abstraction to > replace the copied loopingcall classes. Those loopingcall classes > that were declared in oslo.vmware have been switched to use the > oslo.service abstraction. > > Walt > > > On Tue, Feb 11, 2025 at 4:59 AM Herve Beraud <hberaud(a)redhat.com> > wrote: > > Today we wanted to share with you the latest updates on our > ongoing efforts to phase out Eventlet across our various > OpenStack projects. Below you’ll find a summary of our > progress, key milestones, and important dates to keep in mind > as we continue to move forward with this critical initiative. > > ## Current Status > > The lib freeze is next week and we are close from milestone 3. > At the time I wrote these lines 105 eventlet removal patches > have been submitted on our gerrit and 61 patches are now > merged. That shows really good progress. > > https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22 > > Eventlet is now in version 0.39.0 and is compatible and ready > to be used with Python 3.13, meaning that we are ready for an > upgrade of our runtimes within the coming series. > > ## Achievements > > Among all the contributions, we can highlight these achievements: > > - 2 deliverables are now completely migrated: > python-manilaclient and mistral-lib; > - Neutron implemented their new socket server in replacement > of the eventlet WSGI features and they started to migrate > their OVN agent other that new server; > - Mistral has made excellent progress and significantly > reduced their usages of Eventelt. From moving from coroutines > to threading, or migrating their messaging executor. We will > been soon able to consider mistral as fully migrated; > - Glance started a wide clean up of Eventlet into their > functional tests; > - oslo.service's backend mechanism is now implemented and the > Eventlet isolation as been merged, opening the door of its > refactoring based on threading; > - Almost all Oslo deliverables (apart oslo.vmware) deprecated > Eventlet from their usages; > - Eventlet's support of Python 3.13 has been implemented and > released; > - Eventlet stabilization works have been done during the past > months and we now observe less negative feedback and less new > bugs. No news, good news; > - The community goal is now selected, officializing the > starting of this initiative. > > ## Latest Updates > > There are side conversations concerning the > deprecation/removal of oslo.rootwrap and oslo.vmware. Those > both deliverables rely on Eventlet and are in some way > considered as deprecated or experimental. > > In parallel to that, we think we should also have a discussion > about heat and heat-templates. The activity of these projects > seems quite slow and (do not hesitate to correct me if I'm > wrong) they are more or less in a way to be replaced with more > modern ways to orchestrate OpenStack. Should we not prioritize > their deprecation or at least start a wider discussion about > their fate? > > ## Call to Action > > Epoxy is a SLURP series, and as we are close from lib freeze > and from milestone 3, that's the right time to prioritize your > deprecations in your deliverables in view of removing these > pieces with non-SLURP series. The sooner they are deprecated > the better. > > ## Important Dates > > Please do not hesitate to join us during our bi-weekly meeting > on #openstack-eventlet-removal. > Bring your own topics and do not hesitate to ask questions. > > All the details are available in the links below: > - > https://wiki.openstack.org/wiki/Eventlet-removal#Recurring_Events > - https://etherpad.opendev.org/p/epoxy-eventlet-tracking > > ## Final Thoughts > > This series strongly demonstrated the feasibility to remove > our dependency to Eventlet. There is still a long way to go > but the path is now well cleared. Do not hesitate to take > example on the ongoing efforts, and If there are any questions > or if you need assistance with your project’s transition away > from Eventlet, please join the discussion on the > #openstack-eventlet-removal OFTC channels . We will be happy > to help you! > > Thank you for your dedication and hard work! > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ >

5 months, 2 weeks

Re: [eventlet-removal]When to drop eventlet support

by Jay Faulkner

I'm confused a bit -- the implementation details of our threading modules are not a public API that we owe deprecation periods for. Why are we treating it as such? -JayF On Fri, Jun 13, 2025 at 11:02 AM Sean Mooney <smooney(a)redhat.com> wrote: > > On 13/06/2025 16:55, Ghanshyam Maan wrote: > > ---- On Fri, 13 Jun 2025 08:33:25 -0700 Jay Faulkner <jay(a)gr-oss.io> > wrote --- > > > > > > On 6/13/25 5:08 AM, Balazs Gibizer wrote: > > > > Hi Stackers! > > > > > > > > I would like to sync about the planned timeline of dropping > eventlet > > > > support from OpenStack / Oslo. > > > > > > > > Nova definitely needs at least the full 2026.1 cycle to have a > chance > > > > to transform the nova-compute service. But this plan already feels > > > > stretched based on the progress in the current cycle. So being > > > > conservative means we need the 2026.2 cycle as a buffer. > > > > > > > > Nova would like to keep a release where we support both eventlet > and > > > > threading in parallel. So that operators can do the switching from > > > > eventlet to threading outside of the upgrade procedure. (This was > an > > > > explicit request from them during the PTG). So 2026.2 could be that > > > > version where nova fully supports both concurrency mode, while > > > > eventlet can be marked deprecated. Then the 2027.1 release could be > > > > the first release dropping eventlet. > > > > > > > > However we need to align with the SLURP upgrade as well. 2026.1 is > a > > > > SLURP. But in that release Nova might not be ready to have all > > > > services running in threading mode. So the 2026.1 - 2027.1 SLURP > > > > upgrade would force the operators to change the concurrency mode > > > > during the upgrade itself. > > yes i think that how it should work if an only if a givne service is > capable of > > working without eventlet in the prior slurp. > > if we dont get to the point of being able to run in threaded mode for a > given service > > in 2026.1 then the deprecate of eventlet for tha service will have to be > released in the next slurp > (2027.1) before the removal of support can be done in the next non slupr > 2027.2 > > > > > > > > > I see two ways forward: > > > > * A) We say that operators who want to do the concurrency mode > change > > > > outside of an upgrade could not skip the 2026.2 release, i.e. they > > > > cannot do SLURP directly from 2026.1. to 2027.1. > > > > This has a big impact on upgrades and breaks our SLURP model. > > > > > > * B) We keep supporting the eventlet mode in the 2027.1 release as > > > > well and only dropping support in 2028.1. > > no we can support it in 2027.1 with all serives defaulting ot not using > it if > > its congurable and deprecate it then remove in 2027.2 > > we do not need to extned to 2028.1 > > > > > I am in favour of this option. > > > > I was reading the goal doc about the timeline and found something in > > 'Completion Criteria' section[1] which says: > > - (2027.1) Get usage of Eventlet in oslo deliverables removed; > > so in my mind 2027.1 was either gonig to be the first release without > eventlet support > > or the last release to supprot eventlet mode.(if supported in 2027.1 > eventlet mode would not be the default mode > > and would be deprecated for removal in 2027.2. > > my expectation is the first release of nova to support running in thread > mode would be 2026.1 > but im not convinced we will be confident enough to switch threaded mode > on by default > for all nova services in 2026.1. basically i think it may be funcitonal > but have some bugs or performace issues > that will be refined in 2026.2 wehre we will look to change the default. > > then in 2027.1 we will either start removing the option to run in > eventlet mode > > or deprecate that and remove it in 2027.2 > > my logic is this if we can run in threaded mode in 2026.1 in all service > we will likely deprecate > > eventlet mode in 2026.1, but im not sure we will change the default and > deprecate in the same release. > > > nova may decied to take a per bindary (api vs schduler) approch to > changing the default and doign the eventlet > > removal too but we have not discussed that in detail. > > for example if the nova-schduler is able to run in thread mode this > cycle. we could deprecate runing > it in eventlet mdoe in 2026.1 and make threaded mode the default however > we will still need eventlet for nova-compute. > > so we have a choice to make, only deprecate using eventlet in nova once > all nova service are ready > > or do it per sevice as each part is ready. > > > > > - "(2027.2) Get Eventlet retired from OpenStack;" > > > Again, 2027.2 (non-SLURP) is mentioned as eventlet retirement, I do not > know > > if any technical reason to do it in non-SLURP or it can be moved to > SLURP release. > > Maybe hberaud knows. > > i dont see doing this in a non slurp as a problem under our upgrade policy. > > once we have supprot for runnign without it in a preor slurp even if its > not the defautl we > > are free to deprecate eventlet supprot in that slurp and remove it in a > non slurp. > > to me this jsut reads as we deprecate in 2027.1 annoching it will be > removed in a future release > > adn then we do the removal in the non slurp > > i think doing the actual removal in a non slurp is better then doing it > in a slurp. > > i.e. the deprecations happen in the slurps, the changes of defaults and > remvoal of code happens in the non slurps. > > operators who want to move fast or use newer python release can adopt > the new defaults earlier in the non sluprts > > those that want to move slow get longer for the new mode to bake by > skiping the release and getting the concuracy > > model change on upgrade. > > > > > Anyway, thanks gibi for bringing this. There are many projects that have > not started the > > work yet (Nova might have more work compared to others), but I think we > should discuss/re-discuss > > the timelines considering all projects/challenges. Accordingly, update > the goal doc for the exact timelines > > and what the impact will be for projects that do not finish work as per > the timelines (for example upgrade > > issue, workaround etc). > > > > [1] > https://governance.openstack.org/tc/goals/selected/remove-eventlet.html#com… > > > > -gmaan > > > > > > > > Keeping eventlet running for that long is not something that is a > worthy > > > investment of time. The oslo libraries are showing a deprecation of > > > 2026.2, I've been using that date as the target for all Ironic work > as > > > well. > > > > > > Beyond the oslo team (who I don't speak for), there are folks -- like > > > Itamar on behalf of GR-OSS -- who are doing work behind the scenes to > > > keep eventlet running - barely. I do not expect the GR-OSS > investment in > > > this work to extend much past the midpoint of 2026. > > > > > > > > > My $.02, > > > > > > Jay Faulkner > > > Open Source Developer > > > G-Research Open Source Software > > > > > > > > > >

2 months

Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary

by Herve Beraud

Le jeu. 17 avr. 2025 à 19:07, Clay Gerrard <clay.gerrard(a)gmail.com> a écrit : > > - Swift: Evaluating alternatives with impressive performance results > (FastWsgi showing 10x better performance!). > > A swift core maintainer did a "hello world" application benchmark > comparison of eventlet & fast-wsgi and reported something similar: > > FastWSGI server is about ~10 times faster than Eventlet WSGI server > > AFAIK it's non-trivial and unproven if anything like that comparison would > hold up in a relatively complex application that's actually making backend > requests to other http/rest APIs or talking to ancillary services like > memcache and auth systems inline w/ every request/response. > This mailing thread is just a summary of all the things that have been said about the Eventlet removal during the all sessions of the PTG, so this sentence is just a quote of the same thing you refer to. You can find further details here https://removal.eventlet.org/guide/openstack/flamingo-ptg/ > > IIUC, FastWSGI is using "one thread per request" (actually re-reading the > code that's no longer obvious to me, it's using libuv to handle connections > and parse sockets - but are all call_wsgi_application just blocking!?). > IMHO on modern systems a WSGI server using "one thread per request" is > probably reasonable for the kind of concurrency a swift proxy server > running CPU intensive operations like encryption and erasure coding (and > even decoding JWTs) will want to handle (i.e scale wide, more proxies doing > less concurrent RPS, e.g. with 100s of proxy-application worker nodes we > turn down max_clients from default 1000 to closer to 100 as part of a > solution to provide some back-pressure against overly aggressive concurrent > request spikes), and on the storage nodes OTOH one-thred-per-request would > probably *help* provide more *consistent* performance in the face of full > disks hitting seeks and flush when their blocking filesystem and sqlite > database calls are starving the hub anyway. > > The "hardest" part of a swift migration off eventlet from my perspective > is the gratuitous use of lightweight concurrency to talk to multiple > backend services *within a request* - i.e. each request greenlet > spawns/waits on *multiple* greenlets (think trio nursery) like "connect to > 12 backend servers and stream each EC chunk for every client segment > concurrently" or "start a metadata server update request while waiting on a > thread doing an fsync and if either errors add a entry to a repair journal > before responding" using custom wrappers around GreenPool like a > GreenAsyncPile > https://github.com/NVIDIA/swift/blob/master/swift/common/utils/__init__.py#… > > Can a FastWSGI one-thread-per-request server still share a > ThreadPoolExecutor if you want to offload waiting on a blocking call while > running other coroutines talking to non-blocking APIs like a socket w/i > each request thread? I don't think I see why not... > > Honestly the idea of a focused effort to wean a large system like swift > off of eventlet is daunting to say the least; certainly difficult to do > "optionally" or "experimentally" - it seems like once we port some small > backend service like ... idk the account-auditor - even "just" to use some > kind asyncio wrapper ... I think you just want to *delete* any kind of > "optional" monkey patching? Or is there some way towards a magic > translator (awaitlet?) that lets us try turning off "transparent single > thread w/ implicit concurrency" for "experimental multiple threads each > with explicit cooperative concurrency"? IIUC at some point near the bottom > of the layers of translation wrappers (to help avoid plumbing "async def" > and "return await" all the way from your "__main__ asyncio.run" to the very > bottom of your app) - you DO *eventually* get to the bottom and have to > s/from eventlet.green.http.client import HTTPConnection/from some_async_lib > import AwaitableHTTPConnectionLike/ *right*??? LIke a minimum you have to > "__main__ s/monkey_patch/asyncio.run/" - then you get to play games where > you can say "return awaitlet" instead of every function being a > generator/coroutine ... but whereas "chunk = sock.read()" would "just" > block the greenlet you'd spawned while "doing other stuff in that same > request" we do now actually have to write *some* async def coroutines? Or > maybe at the top you "if BETTER_CONCURRENCY: asyncio.run else monkey_patch" > and at the bottom do "return AwaitableConnection if BETTER_CONCURRENCY else > AwaitableLookingButActuallySomehowJustBoringBustedGreenTransparentBlockingConnection"??? > > I'd love to do some targeted experiments to get a feel for the options; > I'm not sure where to start! > > On Thu, Apr 17, 2025 at 10:44 AM Herve Beraud <hberaud(a)redhat.com> wrote: > >> Following the intensive discussions at the April 2025 Project Teams >> Gathering about the Eventlet removal initiative, this thread summarizes the >> current status, challenges, and strategies of all the OpenStack teams with >> a transversal perspective. >> >> A full and detailed report is available at: >> https://removal.eventlet.org/guide/openstack/flamingo-ptg/ >> >> ## Python 3.13 Compatibility >> >> The countdown has begun! As documented in the April 2025 PTG discussions, >> the OpenStack community is accelerating efforts to remove Eventlet >> dependencies across all projects. This initiative has become critical due >> to Eventlet's compatibility problems with Python 3.13 and the upcoming >> "GILectomy" (PEP 703). >> >> With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing >> a hard deadline for this work >> >> https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 >> >> ## Migration Status >> >> ### Significant Progress >> - Octavia: Fully migrated since 2017! Their approach is now documented as >> a community case study. >> - Mistral: Almost there with a comprehensive migration approach. >> - Neutron: Significant progress with numerous patches already merged. >> - Glance: Can be deployed without Eventlet, though some optional features >> still need work. >> >> ### Work in Progress >> - Nova: Planning a service-by-service migration with dual-mode support >> during transition. >> - Swift: Evaluating alternatives with impressive performance results >> (FastWsgi showing 10x better performance!). >> - Manila, Cinder, Heat: All have active plans for the Flamingo cycle. >> - Designate, Blazar, Watcher: Starting their migration journeys. >> - Ironic: Facing complex challenges, particularly with the Ironic Python >> Agent component. >> >> ## Technical Spotlight: Oslo.service's Threading Backend >> >> A key development is the new threading backend for oslo.service that >> eliminates the Eventlet dependency: >> - Review in progress: >> https://review.opendev.org/c/openstack/oslo.service/+/945720 >> - No longer provides WSGI support >> - Each service will need to deprecate implementations dependent on >> Eventlet's WSGI server >> >> ## Migration Strategies: One Size Doesn't Fit All >> >> Projects are adopting diverse approaches: >> >> - Dual-mode support: Nova and Glance are supporting both Eventlet and >> native threads during transition >> - Canary approach: Swift is considering starting with proxy nodes >> - Component-by-component: Cinder is starting with the Volume Manager >> - Complete replacement: Heat is planning a full discontinuation of WSGI >> server implementations >> >> ## Nova's Detailed Roadmap >> >> Nova has a comprehensive two-cycle plan: >> >> Flamingo Cycle (2025.2): >> - API modernization >> - Architecture updates with environment variable controls >> - Performance improvements >> - Test transition to native threading >> >> Guppy Cycle (2026.1): >> - Core event loop conversion >> - Adoption of oslo.service's new threading backend >> >> ## Issues to Watch: RabbitMQ Heartbeat Problems >> >> The problem with RabbitMQ heartbeats story: >> - Timeouts and API failures in "green" environments >> - Partial solution using pthreads exists but has logging issues >> - Track the fix: https://review.opendev.org/c/openstack/oslo.log/+/937729 >> >> ## How to Get Involved >> >> - Join the #openstack-eventlet-removal channel on OFTC >> - Review the official guide: https://removal.eventlet.org/ >> - Look for patches under the "eventlet-removal" topic: >> https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22 >> >> ## Recommended Reading >> >> For teams starting their migration: >> 1. Official goal documentation: >> https://governance.openstack.org/tc/goals/selected/remove-eventlet.html >> 2. Migration preparation guide: >> https://removal.eventlet.org/guide/preparing-for-migration/ >> 3. Octavia case study: >> https://removal.eventlet.org/guide/case-studies/octavia/ >> >> ## Stay Updated >> >> - The complete version of this report is available online at: >> https://removal.eventlet.org/guide/openstack/flamingo-ptg/ >> - Track the progress of the migration across all projects at: >> https://removal.eventlet.org/guide/openstack/#migration-status >> >> Thanks for reading! >> >> PS: This PTG summary is based on the etherpads following the April 2025 >> PTG discussions (https://ptg.opendev.org/etherpads.html) For more >> details, refer to the full report ( >> https://removal.eventlet.org/guide/openstack/flamingo-ptg/) >> > > > -- > Clay Gerrard > 210 788 9431 > -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud https://github.com/4383/

3 months, 3 weeks

Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary

by Clay Gerrard

> - Swift: Evaluating alternatives with impressive performance results (FastWsgi showing 10x better performance!). A swift core maintainer did a "hello world" application benchmark comparison of eventlet & fast-wsgi and reported something similar: FastWSGI server is about ~10 times faster than Eventlet WSGI server AFAIK it's non-trivial and unproven if anything like that comparison would hold up in a relatively complex application that's actually making backend requests to other http/rest APIs or talking to ancillary services like memcache and auth systems inline w/ every request/response. IIUC, FastWSGI is using "one thread per request" (actually re-reading the code that's no longer obvious to me, it's using libuv to handle connections and parse sockets - but are all call_wsgi_application just blocking!?). IMHO on modern systems a WSGI server using "one thread per request" is probably reasonable for the kind of concurrency a swift proxy server running CPU intensive operations like encryption and erasure coding (and even decoding JWTs) will want to handle (i.e scale wide, more proxies doing less concurrent RPS, e.g. with 100s of proxy-application worker nodes we turn down max_clients from default 1000 to closer to 100 as part of a solution to provide some back-pressure against overly aggressive concurrent request spikes), and on the storage nodes OTOH one-thred-per-request would probably *help* provide more *consistent* performance in the face of full disks hitting seeks and flush when their blocking filesystem and sqlite database calls are starving the hub anyway. The "hardest" part of a swift migration off eventlet from my perspective is the gratuitous use of lightweight concurrency to talk to multiple backend services *within a request* - i.e. each request greenlet spawns/waits on *multiple* greenlets (think trio nursery) like "connect to 12 backend servers and stream each EC chunk for every client segment concurrently" or "start a metadata server update request while waiting on a thread doing an fsync and if either errors add a entry to a repair journal before responding" using custom wrappers around GreenPool like a GreenAsyncPile https://github.com/NVIDIA/swift/blob/master/swift/common/utils/__init__.py#… Can a FastWSGI one-thread-per-request server still share a ThreadPoolExecutor if you want to offload waiting on a blocking call while running other coroutines talking to non-blocking APIs like a socket w/i each request thread? I don't think I see why not... Honestly the idea of a focused effort to wean a large system like swift off of eventlet is daunting to say the least; certainly difficult to do "optionally" or "experimentally" - it seems like once we port some small backend service like ... idk the account-auditor - even "just" to use some kind asyncio wrapper ... I think you just want to *delete* any kind of "optional" monkey patching? Or is there some way towards a magic translator (awaitlet?) that lets us try turning off "transparent single thread w/ implicit concurrency" for "experimental multiple threads each with explicit cooperative concurrency"? IIUC at some point near the bottom of the layers of translation wrappers (to help avoid plumbing "async def" and "return await" all the way from your "__main__ asyncio.run" to the very bottom of your app) - you DO *eventually* get to the bottom and have to s/from eventlet.green.http.client import HTTPConnection/from some_async_lib import AwaitableHTTPConnectionLike/ *right*??? LIke a minimum you have to "__main__ s/monkey_patch/asyncio.run/" - then you get to play games where you can say "return awaitlet" instead of every function being a generator/coroutine ... but whereas "chunk = sock.read()" would "just" block the greenlet you'd spawned while "doing other stuff in that same request" we do now actually have to write *some* async def coroutines? Or maybe at the top you "if BETTER_CONCURRENCY: asyncio.run else monkey_patch" and at the bottom do "return AwaitableConnection if BETTER_CONCURRENCY else AwaitableLookingButActuallySomehowJustBoringBustedGreenTransparentBlockingConnection"??? I'd love to do some targeted experiments to get a feel for the options; I'm not sure where to start! On Thu, Apr 17, 2025 at 10:44 AM Herve Beraud <hberaud(a)redhat.com> wrote: > Following the intensive discussions at the April 2025 Project Teams > Gathering about the Eventlet removal initiative, this thread summarizes the > current status, challenges, and strategies of all the OpenStack teams with > a transversal perspective. > > A full and detailed report is available at: > https://removal.eventlet.org/guide/openstack/flamingo-ptg/ > > ## Python 3.13 Compatibility > > The countdown has begun! As documented in the April 2025 PTG discussions, > the OpenStack community is accelerating efforts to remove Eventlet > dependencies across all projects. This initiative has become critical due > to Eventlet's compatibility problems with Python 3.13 and the upcoming > "GILectomy" (PEP 703). > > With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing a > hard deadline for this work > > https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 > > ## Migration Status > > ### Significant Progress > - Octavia: Fully migrated since 2017! Their approach is now documented as > a community case study. > - Mistral: Almost there with a comprehensive migration approach. > - Neutron: Significant progress with numerous patches already merged. > - Glance: Can be deployed without Eventlet, though some optional features > still need work. > > ### Work in Progress > - Nova: Planning a service-by-service migration with dual-mode support > during transition. > - Swift: Evaluating alternatives with impressive performance results > (FastWsgi showing 10x better performance!). > - Manila, Cinder, Heat: All have active plans for the Flamingo cycle. > - Designate, Blazar, Watcher: Starting their migration journeys. > - Ironic: Facing complex challenges, particularly with the Ironic Python > Agent component. > > ## Technical Spotlight: Oslo.service's Threading Backend > > A key development is the new threading backend for oslo.service that > eliminates the Eventlet dependency: > - Review in progress: > https://review.opendev.org/c/openstack/oslo.service/+/945720 > - No longer provides WSGI support > - Each service will need to deprecate implementations dependent on > Eventlet's WSGI server > > ## Migration Strategies: One Size Doesn't Fit All > > Projects are adopting diverse approaches: > > - Dual-mode support: Nova and Glance are supporting both Eventlet and > native threads during transition > - Canary approach: Swift is considering starting with proxy nodes > - Component-by-component: Cinder is starting with the Volume Manager > - Complete replacement: Heat is planning a full discontinuation of WSGI > server implementations > > ## Nova's Detailed Roadmap > > Nova has a comprehensive two-cycle plan: > > Flamingo Cycle (2025.2): > - API modernization > - Architecture updates with environment variable controls > - Performance improvements > - Test transition to native threading > > Guppy Cycle (2026.1): > - Core event loop conversion > - Adoption of oslo.service's new threading backend > > ## Issues to Watch: RabbitMQ Heartbeat Problems > > The problem with RabbitMQ heartbeats story: > - Timeouts and API failures in "green" environments > - Partial solution using pthreads exists but has logging issues > - Track the fix: https://review.opendev.org/c/openstack/oslo.log/+/937729 > > ## How to Get Involved > > - Join the #openstack-eventlet-removal channel on OFTC > - Review the official guide: https://removal.eventlet.org/ > - Look for patches under the "eventlet-removal" topic: > https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22 > > ## Recommended Reading > > For teams starting their migration: > 1. Official goal documentation: > https://governance.openstack.org/tc/goals/selected/remove-eventlet.html > 2. Migration preparation guide: > https://removal.eventlet.org/guide/preparing-for-migration/ > 3. Octavia case study: > https://removal.eventlet.org/guide/case-studies/octavia/ > > ## Stay Updated > > - The complete version of this report is available online at: > https://removal.eventlet.org/guide/openstack/flamingo-ptg/ > - Track the progress of the migration across all projects at: > https://removal.eventlet.org/guide/openstack/#migration-status > > Thanks for reading! > > PS: This PTG summary is based on the etherpads following the April 2025 > PTG discussions (https://ptg.opendev.org/etherpads.html) For more > details, refer to the full report ( > https://removal.eventlet.org/guide/openstack/flamingo-ptg/) > -- Clay Gerrard 210 788 9431

3 months, 3 weeks

Re: #eventlet-removal - OFTC IRC meeting time slot poll - Express your choice!

by Herve Beraud

The selected time slot is Tuesday 3pm UTC, each first and third Tuesday of the month. https://framadate.org/rUghO3j6ZkRhKD7m Next meeting is next week (Nov 19), here is the list of our meetings for epoxy: https://etherpad.opendev.org/p/epoxy-eventlet-tracking All coming dates are available here https://wiki.openstack.org/wiki/Eventlet-removal#Save_the_Date Thanks for your time Le mer. 6 nov. 2024 à 13:00, Herve Beraud <hberaud(a)redhat.com> a écrit : > Voting will end the day after tomorrow. Many people already expressed > their voice, and you? > Last chance to pick your prefered time slot: > https://framadate.org/rUghO3j6ZkRhKD7m > > Le mer. 30 oct. 2024 à 11:23, Herve Beraud <hberaud(a)redhat.com> a écrit : > >> Hey Stackers, >> >> We propose to schedule a one hour OFTC IRC meeting >> (#opentack-eventlet-removal) each first and third week of each month. >> >> Let's find together the right UTC time slot for our one hour OFTC IRC >> meeting. Once the vote is done, meetings will be scheduled each first and >> third <selected time slot in the poll> of each month. >> >> Please, vote here: https://framadate.org/rUghO3j6ZkRhKD7m >> >> Votes will be considered as closed by Nov. 8th >> >> First meeting will be forced to 4pm UTC on Tuesday Nov 5th and then moved >> to the selected time slot for all the following meetings. See the proposed >> time slots in the poll link above. >> >> Join us on the #opentack-eventlet-removal OFTC IRC channel. >> >> Thanks in advance for your votes >> -- >> Hervé Beraud >> Senior Software Engineer at Red Hat >> irc: hberaud >> https://github.com/4383/ >> >> > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

9 months

Re: Upstream Oslo IRC Meetings Are Back.

by Takashi Kajinami

I noticed that eventlet removal meeting is independent from the oslo meeting proposed in this thread. I'm ok to have dedicated slot to discuss oslo specific topics but may need to have clear scope definition. (Since it was mentioned that eventlet removal is the core topic we want to discuss in this meeting, right ?). Regarding the time slot, EMEA/NA slots might work better for me because I can join these after I settle down my own things, but I think it's generally got to have more Asia friendly slots if anyone from Asia (especially east-Asia) is interested in joining us. On 10/31/24 7:07 PM, dbengt(a)redhat.com wrote: > > > On 10/30/24 5:57 PM, Ghanshyam Mann <gmann(a)ghanshyammann.com> wrote: >> ++ on alternate TZ meetings, and that way, EU and Asia TZ can be covered in one of the meetings. >> To keep it simple, we can have separate doodle poll for each TZ meeting. > Ok so maybe instead we can have one meeting per week. As you proposed like one for the Europeean and Asia and one for the us? I have create a first UTC pool here[1] afternoon and here[2] you have the morning one. > > [1] https://www.vyte.in/e/1cnp51e4sd [2] https://www.vyte.in/e/2j0to257cd >

9 months, 1 week

Re: #eventlet-removal - Progress & Updates - Flamingo milestone 1

by Rodolfo Alonso Hernandez

Hello all: This is the Neutron etherpad: https://etherpad.opendev.org/p/neutron-eventlet-deprecation (we used "deprecation" because we didn't have the tag "eventlet-removal" when we created it) So far, we have achieved the following tasks: * Remove the usage of eventlet in the Neutron API (not the maintenance method, periodic or RPC). Right now, the API only works with WSGI module. * MIgrate the following agents: ** OVN agent ** OVN metadata agent ** Metadata agent ** SRIOV agent The OVS agent, L3 agent and DHCP are the next ones to be migrated and are very close. We are actually expecting the oslo.service new backend ("threading") to continue with these agents removal. Regards. On Wed, May 21, 2025 at 10:20 AM Herve Beraud <hberaud(a)redhat.com> wrote: > Thanks Douglas. > I also added your etherpad into the teams efforts tracker > https://wiki.openstack.org/wiki/Eventlet-removal#Teams_efforts > > Le mar. 20 mai 2025 à 21:02, Douglas Viroel <viroel(a)gmail.com> a écrit : > >> Hi hberaud o/ >> >> Thanks for the updates. >> The Watcher team also started investigating eventlet usage in the code >> and proposed some initial actions. >> We started to follow up this effort in our weekly meeting too. >> Our etherpad is the following: >> https://etherpad.opendev.org/p/watcher-eventlet-removal >> >> BR, >> >> On Tue, May 20, 2025 at 3:51 PM Herve Beraud <hberaud(a)redhat.com> wrote: >> >>> >>> >>> Le mar. 20 mai 2025 à 18:47, Dmitry Tantsur <dtantsur(a)protonmail.com> a >>> écrit : >>> >>>> Hi! >>>> >>>> Thank you for the update, here is the etherpad I created for Ironic: >>>> https://etherpad.opendev.org/p/ironic-eventlet-removal >>>> >>>> >>> Thank you Dimitry, that looks pretty amazing, kudos! Let me add it to >>> our collection of efforts. >>> >>> >>>> Dmitry >>>> >>>> On 5/20/25 6:44 PM, Herve Beraud wrote: >>>> > TL;DR: We just passed the milestone 1 of Flamingo and at this point >>>> big >>>> > fishes like Neutron, Nova, Glance, and Mistral demonstrated awesome >>>> > progress in their migration. Oslo.service's new backend is now usable >>>> in >>>> > real life use cases. Eventlet still suffers from a new version of >>>> > Python. If you have not yet started your migration yet, then, please >>>> > consider starting ASAP before the aircraft crash. >>>> > >>>> > ## OpenStack Updates >>>> > >>>> > Neutron's L3 agent is well on its way to running its WSGI features >>>> > without using Eventlet and activating the new oslo.service threading >>>> > backend at runtime. Allowing to decouple a bit more the needs of >>>> > Eventlet at runtime. >>>> > >>>> > Nova scheduler is now able to start in threading mode without >>>> requiring >>>> > any usage of Eventlet at runtime. This was possible by also shifting >>>> > nova-scheduler to the threading backend of oslo.service. More details >>>> > are available here https://gibizer.github.io/categories/eventlet/ >>>> > <https://gibizer.github.io/categories/eventlet/> >>>> > >>>> > Activating the new threading backend of oslo.service required >>>> devstack >>>> > adaptation to allow having jobs running this execution mode. https:// >>>> > review.opendev.org/c/openstack/devstack/+/948436 <https:// >>>> > review.opendev.org/c/openstack/devstack/+/948436>. Everybody will be >>>> > able to benefit from these adaptations. >>>> > >>>> > Glance continues its adaptation journey by migrating its functional >>>> jobs >>>> > to reduce its needs of Eventlet and to allow it to run jobs in >>>> threading >>>> > mode. >>>> > >>>> > oslo.service's new backend is on the way to be landed soon, and is >>>> > waiting for additional reviews. >>>> https://review.opendev.org/c/openstack/ >>>> > oslo.service/+/945720 <https://review.opendev.org/c/openstack/ >>>> > oslo.service/+/945720> >>>> > >>>> > ## Eventlet Updates >>>> > >>>> > Our CI was failing for almost 2 months, we were able to unlock this >>>> > failing CI and hence we were able to merge several blocked reviews. >>>> > >>>> > Last week Eventlet 0.40.0 was released. This new version provides >>>> fixes >>>> > around OpenSSL 3.5+. >>>> > The support of Python 3.8 and of PyPy were removed. >>>> > >>>> > The Python 3.13 issues are still not solved and require further work. >>>> > >>>> > ## Call to Action >>>> > >>>> > Flamingo is a crucial series in the realization of global migration >>>> of >>>> > OpenStack. We need your involvement to accelerate the progress of >>>> this >>>> > initiative! >>>> > >>>> > If your project is affected by Eventlet, now is the time to start >>>> your >>>> > migration. Teams are invited to read the migration guide available at >>>> > https://removal.eventlet.org/ <https://removal.eventlet.org/> and >>>> are >>>> > encouraged to join the #openstack-eventlet-removal IRC channel to get >>>> help. >>>> > >>>> > You'll benefit quickly from the advantages of running your >>>> deliverables >>>> > without using Eventlet. >>>> > >>>> > ## Final Thoughts >>>> > >>>> > Postponing the beginning of the migration will put additional >>>> pressure >>>> > related to growing runtime failures of Eventlet. We want to prevent >>>> this >>>> > from happening to you. >>>> > >>>> > We are now going straight to milestone 2 meaning that we will soon be >>>> in >>>> > the middle of Flamingo. >>>> > >>>> > The migration path is now well cleared by all the previous actions >>>> and >>>> > changes applied into OpenStack. Do not be afraid by the migration and >>>> > take advantage of the collected experience. >>>> > >>>> > Your journey start here https://removal.eventlet.org/ <https:// >>>> > removal.eventlet.org/> >>>> > >>>> > Thanks for reading. >>>> > >>>> > -- >>>> > Hervé Beraud >>>> > Principal Software Engineer at Red Hat >>>> > irc: hberaud >>>> > https://github.com/4383/ <https://github.com/4383/> >>>> > >>>> >>>> >>>> >>> >>> -- >>> Hervé Beraud >>> Principal Software Engineer at Red Hat >>> irc: hberaud >>> https://github.com/4383/ >>> >>> >> >> -- >> Douglas Viroel - dviroel >> > > > -- > Hervé Beraud > Principal Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > >

2 months, 2 weeks

[heat][ptg][ops] Flamingo PTG summary

by Takashi Kajinami

Hello, Last week during the PTG the heat team gathered to discuss a few topics. Thank you all who attended the session and the valuable discussions. The details can be found in https://etherpad.opendev.org/p/apr2025-ptg-heat but I'll leave some summary. There are a few plans about which we want to ask for any feedback from operators so I'm adding [ops] tag here. 1. Eventlet removal During the session we reviewed the current global plan to remove/replace eventlet and discussed the plan to follow it in heat. Our current idea is to remove/deprecate unmaintained code first so that we can shrink the scope. At this moment we aim to deprecate and remove the following items. - Standalone heat-api and heat-cfn-api - These are deprecated due to deprecation of WSGI server implementation from oslo.service, which depends on eventlet. Users can run api server by external server mechanism such as mod_wsgi or uwsgi. - heat-all service - This is deprecated due to deprecation of WSGI server implementation from oslo.service, which depends on eventlet. We expect this is not really used in production - legacy engine - This was deprecated in favor of convergence. Note that removing this means we no longer support adopt/abandon/check feature. Question to operators: Does anyone have concern about deprecating/removing these items ? Please be aware that the current heat team is quite small and we need actual help to replace eventlet for these features. 2. CFN API We discussed the current status of heat-cfntool which heavily depends on unmaintained boto library. However we also agreed that replacing boto by boto3 requires some effort due to the fact that boto3 does not allow overriding endpoint quite easily. While we aim to deprecate CFN api, we agreed that we still have to asses the impact because CFN API is used for signaling by default. Question to operators: Is anyone using CFN API in any other use case than signaling ? For the first topic, if we don't hear any objections, we'll follow the plan to move the removal work forward. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

3 months, 4 weeks

Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary

by Thomas Goirand

Hi, On 4/17/25 17:44, Herve Beraud wrote: > With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing > a hard deadline for this work > > https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 If I may: Debian 13 (aka: Trixie) is already in (soft) freeze, and shipping Python 3.13 only also. Harder freeze is in a month, complete freeze is still not announced yet. I'm also very worried about the state of Eventlet with Epoxy on Trixie (I haven't had time to tempest test it yet). Is there a new eventlet release that's going to fix more issues? Will you continue working on the patch you pointed at, and make sure OpenStack continues to work with it? Thanks for all your work on this, Cheers, Thomas Goirand (zigo)

3 months, 3 weeks

openstack-discuss search results for query "#eventlet-removal"