Search results for "#eventlet-removal" - openstack-discuss

Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary

by Clay Gerrard

> - Swift: Evaluating alternatives with impressive performance results (FastWsgi showing 10x better performance!). A swift core maintainer did a "hello world" application benchmark comparison of eventlet & fast-wsgi and reported something similar: FastWSGI server is about ~10 times faster than Eventlet WSGI server AFAIK it's non-trivial and unproven if anything like that comparison would hold up in a relatively complex application that's actually making backend requests to other http/rest APIs or talking to ancillary services like memcache and auth systems inline w/ every request/response. IIUC, FastWSGI is using "one thread per request" (actually re-reading the code that's no longer obvious to me, it's using libuv to handle connections and parse sockets - but are all call_wsgi_application just blocking!?). IMHO on modern systems a WSGI server using "one thread per request" is probably reasonable for the kind of concurrency a swift proxy server running CPU intensive operations like encryption and erasure coding (and even decoding JWTs) will want to handle (i.e scale wide, more proxies doing less concurrent RPS, e.g. with 100s of proxy-application worker nodes we turn down max_clients from default 1000 to closer to 100 as part of a solution to provide some back-pressure against overly aggressive concurrent request spikes), and on the storage nodes OTOH one-thred-per-request would probably *help* provide more *consistent* performance in the face of full disks hitting seeks and flush when their blocking filesystem and sqlite database calls are starving the hub anyway. The "hardest" part of a swift migration off eventlet from my perspective is the gratuitous use of lightweight concurrency to talk to multiple backend services *within a request* - i.e. each request greenlet spawns/waits on *multiple* greenlets (think trio nursery) like "connect to 12 backend servers and stream each EC chunk for every client segment concurrently" or "start a metadata server update request while waiting on a thread doing an fsync and if either errors add a entry to a repair journal before responding" using custom wrappers around GreenPool like a GreenAsyncPile https://github.com/NVIDIA/swift/blob/master/swift/common/utils/__init__.py#… Can a FastWSGI one-thread-per-request server still share a ThreadPoolExecutor if you want to offload waiting on a blocking call while running other coroutines talking to non-blocking APIs like a socket w/i each request thread? I don't think I see why not... Honestly the idea of a focused effort to wean a large system like swift off of eventlet is daunting to say the least; certainly difficult to do "optionally" or "experimentally" - it seems like once we port some small backend service like ... idk the account-auditor - even "just" to use some kind asyncio wrapper ... I think you just want to *delete* any kind of "optional" monkey patching? Or is there some way towards a magic translator (awaitlet?) that lets us try turning off "transparent single thread w/ implicit concurrency" for "experimental multiple threads each with explicit cooperative concurrency"? IIUC at some point near the bottom of the layers of translation wrappers (to help avoid plumbing "async def" and "return await" all the way from your "__main__ asyncio.run" to the very bottom of your app) - you DO *eventually* get to the bottom and have to s/from eventlet.green.http.client import HTTPConnection/from some_async_lib import AwaitableHTTPConnectionLike/ *right*??? LIke a minimum you have to "__main__ s/monkey_patch/asyncio.run/" - then you get to play games where you can say "return awaitlet" instead of every function being a generator/coroutine ... but whereas "chunk = sock.read()" would "just" block the greenlet you'd spawned while "doing other stuff in that same request" we do now actually have to write *some* async def coroutines? Or maybe at the top you "if BETTER_CONCURRENCY: asyncio.run else monkey_patch" and at the bottom do "return AwaitableConnection if BETTER_CONCURRENCY else AwaitableLookingButActuallySomehowJustBoringBustedGreenTransparentBlockingConnection"??? I'd love to do some targeted experiments to get a feel for the options; I'm not sure where to start! On Thu, Apr 17, 2025 at 10:44 AM Herve Beraud <hberaud(a)redhat.com> wrote: > Following the intensive discussions at the April 2025 Project Teams > Gathering about the Eventlet removal initiative, this thread summarizes the > current status, challenges, and strategies of all the OpenStack teams with > a transversal perspective. > > A full and detailed report is available at: > https://removal.eventlet.org/guide/openstack/flamingo-ptg/ > > ## Python 3.13 Compatibility > > The countdown has begun! As documented in the April 2025 PTG discussions, > the OpenStack community is accelerating efforts to remove Eventlet > dependencies across all projects. This initiative has become critical due > to Eventlet's compatibility problems with Python 3.13 and the upcoming > "GILectomy" (PEP 703). > > With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing a > hard deadline for this work > > https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 > > ## Migration Status > > ### Significant Progress > - Octavia: Fully migrated since 2017! Their approach is now documented as > a community case study. > - Mistral: Almost there with a comprehensive migration approach. > - Neutron: Significant progress with numerous patches already merged. > - Glance: Can be deployed without Eventlet, though some optional features > still need work. > > ### Work in Progress > - Nova: Planning a service-by-service migration with dual-mode support > during transition. > - Swift: Evaluating alternatives with impressive performance results > (FastWsgi showing 10x better performance!). > - Manila, Cinder, Heat: All have active plans for the Flamingo cycle. > - Designate, Blazar, Watcher: Starting their migration journeys. > - Ironic: Facing complex challenges, particularly with the Ironic Python > Agent component. > > ## Technical Spotlight: Oslo.service's Threading Backend > > A key development is the new threading backend for oslo.service that > eliminates the Eventlet dependency: > - Review in progress: > https://review.opendev.org/c/openstack/oslo.service/+/945720 > - No longer provides WSGI support > - Each service will need to deprecate implementations dependent on > Eventlet's WSGI server > > ## Migration Strategies: One Size Doesn't Fit All > > Projects are adopting diverse approaches: > > - Dual-mode support: Nova and Glance are supporting both Eventlet and > native threads during transition > - Canary approach: Swift is considering starting with proxy nodes > - Component-by-component: Cinder is starting with the Volume Manager > - Complete replacement: Heat is planning a full discontinuation of WSGI > server implementations > > ## Nova's Detailed Roadmap > > Nova has a comprehensive two-cycle plan: > > Flamingo Cycle (2025.2): > - API modernization > - Architecture updates with environment variable controls > - Performance improvements > - Test transition to native threading > > Guppy Cycle (2026.1): > - Core event loop conversion > - Adoption of oslo.service's new threading backend > > ## Issues to Watch: RabbitMQ Heartbeat Problems > > The problem with RabbitMQ heartbeats story: > - Timeouts and API failures in "green" environments > - Partial solution using pthreads exists but has logging issues > - Track the fix: https://review.opendev.org/c/openstack/oslo.log/+/937729 > > ## How to Get Involved > > - Join the #openstack-eventlet-removal channel on OFTC > - Review the official guide: https://removal.eventlet.org/ > - Look for patches under the "eventlet-removal" topic: > https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22 > > ## Recommended Reading > > For teams starting their migration: > 1. Official goal documentation: > https://governance.openstack.org/tc/goals/selected/remove-eventlet.html > 2. Migration preparation guide: > https://removal.eventlet.org/guide/preparing-for-migration/ > 3. Octavia case study: > https://removal.eventlet.org/guide/case-studies/octavia/ > > ## Stay Updated > > - The complete version of this report is available online at: > https://removal.eventlet.org/guide/openstack/flamingo-ptg/ > - Track the progress of the migration across all projects at: > https://removal.eventlet.org/guide/openstack/#migration-status > > Thanks for reading! > > PS: This PTG summary is based on the etherpads following the April 2025 > PTG discussions (https://ptg.opendev.org/etherpads.html) For more > details, refer to the full report ( > https://removal.eventlet.org/guide/openstack/flamingo-ptg/) > -- Clay Gerrard 210 788 9431

8 months, 2 weeks

[watcher] 2nd call for maintainers - MAAS integration

by Douglas Viroel

Hi all, The Watcher team had a session[1] during the last PTG about the impacts of service integrations that have lacked maintainers, documentation, and/or testing coverage in CI. The MAAS (Metal As A Service) integration still lacks maintainers and is impacting the eventlet removal effort within Watcher, since it contains eventlet-dependent code that requires maintenance and validation. In the 2025.2 development cycle, we had a first call for maintainers[2] and we didn't receive any feedback. This is a second call for maintainers interested in supporting MAAS in Watcher, to help fill these gaps (eventlet removal, CI testing, documentation). The Watcher team plans to *deprecate* this integration during the 2026.1 cycle if we don't find anyone to assist in maintaining it. [1] https://etherpad.opendev.org/p/watcher-2026.1-ptg#L177 [2] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… Thanks and regards, -- Douglas Viroel - dviroel

1 month, 2 weeks

#eventlet-removal - Progress & Updates - Flamingo Past Milestone 3

by Herve Beraud

TL;DR: We just recently passed Flamingo's milestone 3 and since the last ML update we observed tremendous progress. Ironic, Barbican, Heat completed their migration. Futurist has been improved with new useful features for threading mode. The migration timeline has been adjusted to match Nova's dual mode support. PTG is under preparation. Statistics show good progress. Eventlet now supports Python 3.14. ## OpenStack Updates From a statistical perspective: - almost 17% of the occurences of the runtime code initially impacted by Eventlet have been tackled — outside of code dedicated to test, doc, releasenotes, etc; - if we include all the spectrum of occurrences to migrate — code, tests, doc, examples — it represent ~14% of the entire code base already migrated; - more than 40% of all impacted repositories have made some progress in link with the migration; - 13 projects have been fully migrated since the beginning; - more than 332 patches related to the eventlet migration have been submitted to gerrit. From the teams perspective, we can highlight Ironic completed its migration, for the occasion wrote an excellent write-up [1] — thanks Julia — I strongly encourage you to read it. Barbican [2] and heat [3] successfully completed their migrations too. They are now fully threading based. Dmitry in collaboration with the Ironic team implemented a DynamicPoolExecutor in Futurist [4]. It helps replace dynamic GreenPool. Oslo.service's backend system is now documented [5]. There is still ongoing work on several OpenStack deliverables: Neutron, Nova, Manila, Magnum, Mistral, Watcher, Designate, Trove, etc. All these teams demonstrated awesome progress. The migration timeline has been adjusted at the governance level to match Nova's dual mode support [6]. There is ongoing preparation of the PTG sessions. I think there will be a short slot during one of the TC sessions, and surely a fully dedicated one hour slot to host cross team open discussions. ## Eventlet Updates The support of Python 3.14 has been introduced within Eventlet 0.40.2. A vulnerability (CVE-2025-58068, CVSS 6.3) in Eventlet’s WSGI parser allowing HTTP Request Smuggling (bypass controls, cache poisoning, user targeting) has been found. It has been fixed within Eventlet 0.40.3. A security advisory has been published [7]. ## Call to Action That would be worth considering introducing small experiments as opportunities to identify or learn any weaknesses in new threading architectural models. That could start by preparing and performing some perf scale testing at your deliverables levels. That will help you to define the right scaling configuration for your deliverables. You can draw inspiration from Neutron's rally jobs [8] and from Ironic's scripting created for benchmark purposes [9]. More than a year ago, Dan sparked the performance testing topic idea [10], that's the right time to consider it more deeply. If you do not start the migration of your deliverables, I'd encourage you to plan and prioritize it as soon as possible. Time flies and the later your migration starts the harder it will be. Python 3.15 comes with its share of breaking changes [11] so I'd not suggest waiting for that time. If you are scheduling a PTG session on your side about Eventlet, then please let us know by updating the eventlet removal wiki page [12]. If your service is now migrated, I'd invite you to tag the launchpad bugs which were related to Eventlet, and, if it makes sense to close them. You are now entering a clean phase. ## Final Thoughts During Flamingo we observed tremendous progress, and we are now going straight toward the final release of Flamingo. The PTG, and Gazpacho that will be the logical continuation of the jobs already done. Flamingo will mark a significant release due to the fully migrated bricks and services. They will allow us to enter a beneficial feedback loop. The migration path is now well cleared by all the previous actions and changes applied into OpenStack. Postponing the beginning of the migration will make your life harder. We want to prevent this from happening to you. Your journey start here https://removal.eventlet.org/ If you need information join-us on the #openstack-eventlet-removal OFTC channel. Thanks to all the people who made this migration possible until now. Thanks for your attention. [1] https://ironicbaremetal.org/blog/coming-soon-threading/ [2] https://review.opendev.org/c/openstack/barbican/+/955301 [3] https://review.opendev.org/c/openstack/heat/+/952651 [4] https://docs.openstack.org/futurist/latest/reference/index.html#futurist.Dy… [5] https://docs.openstack.org/oslo.service/latest/user/backend.html [6] https://review.opendev.org/c/openstack/governance/+/952903 [7] https://github.com/advisories/GHSA-hw6f-rjfj-j7j7 [8] https://opendev.org/openstack/neutron/src/branch/master/rally-jobs [9] https://opendev.org/openstack/ironic/src/branch/master/tools/benchmark [10] https://review.opendev.org/c/openstack/governance/+/902585/comment/979db3f6… [11] https://docs.python.org/es/dev/whatsnew/3.15.html#threading [12] https://wiki.openstack.org/wiki/Eventlet-removal#Project-specific_events -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud https://github.com/4383/

3 months, 4 weeks

Re: #eventlet-removal - OFTC IRC meeting time slot poll - Express your choice!

by Herve Beraud

The selected time slot is Tuesday 3pm UTC, each first and third Tuesday of the month. https://framadate.org/rUghO3j6ZkRhKD7m Next meeting is next week (Nov 19), here is the list of our meetings for epoxy: https://etherpad.opendev.org/p/epoxy-eventlet-tracking All coming dates are available here https://wiki.openstack.org/wiki/Eventlet-removal#Save_the_Date Thanks for your time Le mer. 6 nov. 2024 à 13:00, Herve Beraud <hberaud(a)redhat.com> a écrit : > Voting will end the day after tomorrow. Many people already expressed > their voice, and you? > Last chance to pick your prefered time slot: > https://framadate.org/rUghO3j6ZkRhKD7m > > Le mer. 30 oct. 2024 à 11:23, Herve Beraud <hberaud(a)redhat.com> a écrit : > >> Hey Stackers, >> >> We propose to schedule a one hour OFTC IRC meeting >> (#opentack-eventlet-removal) each first and third week of each month. >> >> Let's find together the right UTC time slot for our one hour OFTC IRC >> meeting. Once the vote is done, meetings will be scheduled each first and >> third <selected time slot in the poll> of each month. >> >> Please, vote here: https://framadate.org/rUghO3j6ZkRhKD7m >> >> Votes will be considered as closed by Nov. 8th >> >> First meeting will be forced to 4pm UTC on Tuesday Nov 5th and then moved >> to the selected time slot for all the following meetings. See the proposed >> time slots in the poll link above. >> >> Join us on the #opentack-eventlet-removal OFTC IRC channel. >> >> Thanks in advance for your votes >> -- >> Hervé Beraud >> Senior Software Engineer at Red Hat >> irc: hberaud >> https://github.com/4383/ >> >> > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 year, 1 month

Re: #eventlet-removal - Progress & Updates - Flamingo milestone 1

by Rodolfo Alonso Hernandez

Hello all: This is the Neutron etherpad: https://etherpad.opendev.org/p/neutron-eventlet-deprecation (we used "deprecation" because we didn't have the tag "eventlet-removal" when we created it) So far, we have achieved the following tasks: * Remove the usage of eventlet in the Neutron API (not the maintenance method, periodic or RPC). Right now, the API only works with WSGI module. * MIgrate the following agents: ** OVN agent ** OVN metadata agent ** Metadata agent ** SRIOV agent The OVS agent, L3 agent and DHCP are the next ones to be migrated and are very close. We are actually expecting the oslo.service new backend ("threading") to continue with these agents removal. Regards. On Wed, May 21, 2025 at 10:20 AM Herve Beraud <hberaud(a)redhat.com> wrote: > Thanks Douglas. > I also added your etherpad into the teams efforts tracker > https://wiki.openstack.org/wiki/Eventlet-removal#Teams_efforts > > Le mar. 20 mai 2025 à 21:02, Douglas Viroel <viroel(a)gmail.com> a écrit : > >> Hi hberaud o/ >> >> Thanks for the updates. >> The Watcher team also started investigating eventlet usage in the code >> and proposed some initial actions. >> We started to follow up this effort in our weekly meeting too. >> Our etherpad is the following: >> https://etherpad.opendev.org/p/watcher-eventlet-removal >> >> BR, >> >> On Tue, May 20, 2025 at 3:51 PM Herve Beraud <hberaud(a)redhat.com> wrote: >> >>> >>> >>> Le mar. 20 mai 2025 à 18:47, Dmitry Tantsur <dtantsur(a)protonmail.com> a >>> écrit : >>> >>>> Hi! >>>> >>>> Thank you for the update, here is the etherpad I created for Ironic: >>>> https://etherpad.opendev.org/p/ironic-eventlet-removal >>>> >>>> >>> Thank you Dimitry, that looks pretty amazing, kudos! Let me add it to >>> our collection of efforts. >>> >>> >>>> Dmitry >>>> >>>> On 5/20/25 6:44 PM, Herve Beraud wrote: >>>> > TL;DR: We just passed the milestone 1 of Flamingo and at this point >>>> big >>>> > fishes like Neutron, Nova, Glance, and Mistral demonstrated awesome >>>> > progress in their migration. Oslo.service's new backend is now usable >>>> in >>>> > real life use cases. Eventlet still suffers from a new version of >>>> > Python. If you have not yet started your migration yet, then, please >>>> > consider starting ASAP before the aircraft crash. >>>> > >>>> > ## OpenStack Updates >>>> > >>>> > Neutron's L3 agent is well on its way to running its WSGI features >>>> > without using Eventlet and activating the new oslo.service threading >>>> > backend at runtime. Allowing to decouple a bit more the needs of >>>> > Eventlet at runtime. >>>> > >>>> > Nova scheduler is now able to start in threading mode without >>>> requiring >>>> > any usage of Eventlet at runtime. This was possible by also shifting >>>> > nova-scheduler to the threading backend of oslo.service. More details >>>> > are available here https://gibizer.github.io/categories/eventlet/ >>>> > <https://gibizer.github.io/categories/eventlet/> >>>> > >>>> > Activating the new threading backend of oslo.service required >>>> devstack >>>> > adaptation to allow having jobs running this execution mode. https:// >>>> > review.opendev.org/c/openstack/devstack/+/948436 <https:// >>>> > review.opendev.org/c/openstack/devstack/+/948436>. Everybody will be >>>> > able to benefit from these adaptations. >>>> > >>>> > Glance continues its adaptation journey by migrating its functional >>>> jobs >>>> > to reduce its needs of Eventlet and to allow it to run jobs in >>>> threading >>>> > mode. >>>> > >>>> > oslo.service's new backend is on the way to be landed soon, and is >>>> > waiting for additional reviews. >>>> https://review.opendev.org/c/openstack/ >>>> > oslo.service/+/945720 <https://review.opendev.org/c/openstack/ >>>> > oslo.service/+/945720> >>>> > >>>> > ## Eventlet Updates >>>> > >>>> > Our CI was failing for almost 2 months, we were able to unlock this >>>> > failing CI and hence we were able to merge several blocked reviews. >>>> > >>>> > Last week Eventlet 0.40.0 was released. This new version provides >>>> fixes >>>> > around OpenSSL 3.5+. >>>> > The support of Python 3.8 and of PyPy were removed. >>>> > >>>> > The Python 3.13 issues are still not solved and require further work. >>>> > >>>> > ## Call to Action >>>> > >>>> > Flamingo is a crucial series in the realization of global migration >>>> of >>>> > OpenStack. We need your involvement to accelerate the progress of >>>> this >>>> > initiative! >>>> > >>>> > If your project is affected by Eventlet, now is the time to start >>>> your >>>> > migration. Teams are invited to read the migration guide available at >>>> > https://removal.eventlet.org/ <https://removal.eventlet.org/> and >>>> are >>>> > encouraged to join the #openstack-eventlet-removal IRC channel to get >>>> help. >>>> > >>>> > You'll benefit quickly from the advantages of running your >>>> deliverables >>>> > without using Eventlet. >>>> > >>>> > ## Final Thoughts >>>> > >>>> > Postponing the beginning of the migration will put additional >>>> pressure >>>> > related to growing runtime failures of Eventlet. We want to prevent >>>> this >>>> > from happening to you. >>>> > >>>> > We are now going straight to milestone 2 meaning that we will soon be >>>> in >>>> > the middle of Flamingo. >>>> > >>>> > The migration path is now well cleared by all the previous actions >>>> and >>>> > changes applied into OpenStack. Do not be afraid by the migration and >>>> > take advantage of the collected experience. >>>> > >>>> > Your journey start here https://removal.eventlet.org/ <https:// >>>> > removal.eventlet.org/> >>>> > >>>> > Thanks for reading. >>>> > >>>> > -- >>>> > Hervé Beraud >>>> > Principal Software Engineer at Red Hat >>>> > irc: hberaud >>>> > https://github.com/4383/ <https://github.com/4383/> >>>> > >>>> >>>> >>>> >>> >>> -- >>> Hervé Beraud >>> Principal Software Engineer at Red Hat >>> irc: hberaud >>> https://github.com/4383/ >>> >>> >> >> -- >> Douglas Viroel - dviroel >> > > > -- > Hervé Beraud > Principal Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > >

7 months, 1 week

Re: Upstream Oslo IRC Meetings Are Back.

by Takashi Kajinami

I noticed that eventlet removal meeting is independent from the oslo meeting proposed in this thread. I'm ok to have dedicated slot to discuss oslo specific topics but may need to have clear scope definition. (Since it was mentioned that eventlet removal is the core topic we want to discuss in this meeting, right ?). Regarding the time slot, EMEA/NA slots might work better for me because I can join these after I settle down my own things, but I think it's generally got to have more Asia friendly slots if anyone from Asia (especially east-Asia) is interested in joining us. On 10/31/24 7:07 PM, dbengt(a)redhat.com wrote: > > > On 10/30/24 5:57 PM, Ghanshyam Mann <gmann(a)ghanshyammann.com> wrote: >> ++ on alternate TZ meetings, and that way, EU and Asia TZ can be covered in one of the meetings. >> To keep it simple, we can have separate doodle poll for each TZ meeting. > Ok so maybe instead we can have one meeting per week. As you proposed like one for the Europeean and Asia and one for the us? I have create a first UTC pool here[1] afternoon and here[2] you have the morning one. > > [1] https://www.vyte.in/e/1cnp51e4sd [2] https://www.vyte.in/e/2j0to257cd >

1 year, 2 months

[heat][ptg][ops] Flamingo PTG summary

by Takashi Kajinami

Hello, Last week during the PTG the heat team gathered to discuss a few topics. Thank you all who attended the session and the valuable discussions. The details can be found in https://etherpad.opendev.org/p/apr2025-ptg-heat but I'll leave some summary. There are a few plans about which we want to ask for any feedback from operators so I'm adding [ops] tag here. 1. Eventlet removal During the session we reviewed the current global plan to remove/replace eventlet and discussed the plan to follow it in heat. Our current idea is to remove/deprecate unmaintained code first so that we can shrink the scope. At this moment we aim to deprecate and remove the following items. - Standalone heat-api and heat-cfn-api - These are deprecated due to deprecation of WSGI server implementation from oslo.service, which depends on eventlet. Users can run api server by external server mechanism such as mod_wsgi or uwsgi. - heat-all service - This is deprecated due to deprecation of WSGI server implementation from oslo.service, which depends on eventlet. We expect this is not really used in production - legacy engine - This was deprecated in favor of convergence. Note that removing this means we no longer support adopt/abandon/check feature. Question to operators: Does anyone have concern about deprecating/removing these items ? Please be aware that the current heat team is quite small and we need actual help to replace eventlet for these features. 2. CFN API We discussed the current status of heat-cfntool which heavily depends on unmaintained boto library. However we also agreed that replacing boto by boto3 requires some effort due to the fact that boto3 does not allow overriding endpoint quite easily. While we aim to deprecate CFN api, we agreed that we still have to asses the impact because CFN API is used for signaling by default. Question to operators: Is anyone using CFN API in any other use case than signaling ? For the first topic, if we don't hear any objections, we'll follow the plan to move the removal work forward. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

8 months, 2 weeks

Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary

by Thomas Goirand

Hi, On 4/17/25 17:44, Herve Beraud wrote: > With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing > a hard deadline for this work > > https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 If I may: Debian 13 (aka: Trixie) is already in (soft) freeze, and shipping Python 3.13 only also. Harder freeze is in a month, complete freeze is still not announced yet. I'm also very worried about the state of Eventlet with Epoxy on Trixie (I haven't had time to tempest test it yet). Is there a new eventlet release that's going to fix more issues? Will you continue working on the patch you pointed at, and make sure OpenStack continues to work with it? Thanks for all your work on this, Cheers, Thomas Goirand (zigo)

8 months, 2 weeks

[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?

by Stephen Finucane

o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/

1 year, 2 months

Re: #eventlet-removal - Progress & Update

by smooney＠redhat.com

On Wed, 2024-10-09 at 12:01 +0200, Sylvain Bauza wrote: > Le ven. 4 oct. 2024 à 17:41, engineer2024 <engineerlinux2024(a)gmail.com> a > écrit : > > > That's encouraging news. I hope the new alternative would support a python > > debugger like pdb. This will help the new users to understand the > > flow better. So when can we expect initial release atleast beta of eventlet > > free nova components ? > > > > > That's an excellent question that I don't have an answer. We started to > work on it on Dalmatian but for the moment, I don't know whether the owner > could continue to work on it on Epoxy or if someone else would do it. i dont think we will ever have a "beta" release without eventlet our expection is that we will do a phased removal. so the iniall goal was to to tackel some of the low hanging furit in dalmation the first part of that was spliting the delivberable in nova that actully need eventlet vs those that dont i.e. the command line clinets were monkey patched but never needed eventlet so those were simple to seprate to remove the monkey patching thats what https://review.opendev.org/c/openstack/nova/+/904424 does the next step is to remove our use of eventlets thread pool and replace it with futureist's version. https://review.opendev.org/c/openstack/nova/+/922497/5 https://review.opendev.org/c/openstack/nova/+/905287/9 https://review.opendev.org/c/openstack/nova/+/917962/8 the third step is to remove the only usage of eventlet in the nova-api wsgi module https://review.opendev.org/c/openstack/nova/+/905284/7 i may or may not have time to work on this in 2025.1 i will likely try to make some progress but i may need to hand this off to someone else or rope in some other to help. if we can compelte what i had orginally scoped in https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 for 2025.1 that woudl be a good first step. next steps would be two fold. first we need to change how spawn core service event loop by using the new backend being added by https://review.opendev.org/c/openstack/oslo-specs/+/927503 second we need to change how we handel rpc requests. when the agent recives an rpc instest of executiong the function on the main thread we need to dispatch that to either a GreenThreadPoolExecutor when we are monkey patched (using eventlet) https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L318 or a ThreadPoolExecutor https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L99 when we are not usign eventlet. that is requried so we can still have concurrent interleaved rpc processing and allow the threaded version to just block waiting on future as needed. that bit i have not figured out exactly where to do that yet. but factoring out the rpc handling to use the executor interface would likely be eventlet-removal-part-2 (2025.2) since that can be done without a dep onthe oslo.service change and then eventlet-removal-part-3 2026.1 woudl be converting to the new oslo.servce backend and any remaining clean up. that is optimistic and a lot of work so it may take longer or we might be able to start some other things earlier. tehre is a lot of good work in https://wiki.openstack.org/wiki/Eventlet-removal#Migration_Plan also. nova at present is not planning to use asynio but that does nto mean we never will we just dont have a high enough concurrence requireemtn to need it so trhead pools shoud be sufficient once we are using them there is no reaosn we coudl not have an AsyncioPoolExecutor using awaitlet or explore using asyncio more directly to your debugger point any servie that is not monkey patched will gain the ablity to just run in a debugger normally i.e. once we get rid of using it in the api wsgi module you should be able to just debug it but keep in mind that nova is a distributed system os if you set a breakpoint you will very likly cause request to time out. debuging a distribute system is not the same as debuging a application running on a single host and never will be. that is not a goal of this work its just a side benifit. > > -Sylvain > > On Fri, 4 Oct 2024, 20:38 Herve Beraud, <hberaud(a)redhat.com> wrote: > > > > > We’re excited to share the latest updates on our ongoing efforts to phase > > > out Eventlet across our various OpenStack projects. Below you’ll find a > > > summary of our progress, key milestones, and important dates to keep in > > > mind as we move forward with this critical initiative. > > > > > > ## Current Status > > > > > > We’ve completed the initial analysis phase across more than 120 OpenStack > > > projects. This includes identifying the various patterns where Eventlet is > > > currently in use and evaluating where it can be safely removed or disabled. > > > Here are some highlights: > > > > > > - Completed Analysis: Projects where Eventlet is used include (but are > > > not limited to): Aodh, Barbican, Nova, Neutron, Tacker, Taskflow, Trove, > > > Venus, Watcher, and Zun. > > > > > > - Eventlet Optional: Several projects have been identified where Eventlet > > > seems to be optional and can be disabled with minimal impact. For these > > > projects, we will move forward the discussion with teams to try > > > deactivating Eventlet in upcoming updates. Once done, projects of this kind > > > would be able to start their migration with more serenity. > > > > > > - Eventlet Required: Some projects still rely heavily on Eventlet for > > > core functionality. For these, we are going to develop specific migration > > > plans. > > > > > > ## Migration Plans in Progress > > > > > > - oslo.db: Preparing its Asyncio based engine facade; > > > - oslo.service: Refactoring plans being discussed to minimize disruption > > > during the migration; > > > - glance: Refactoring code to use native thread model rather than Eventlet > > > > > > More details can be found here: > > > https://review.opendev.org/q/topic:%22eventlet-removal%22 > > > > > > ## Latest Updates > > > > > > - Community Goal Transition: The ongoing community goal for removing > > > Eventlet is now in the process of becoming an official "selected" goal, > > > https://review.opendev.org/c/openstack/governance/+/931254. > > > > > > - Pattern Identification Completed: We’ve identified key patterns related > > > to Eventlet usage, including its common integration with monkey_patch(), > > > eventlet.Timeout, eventlet.GreenPool, and eventlet.wsgi.server(). Visit the > > > official wiki to get more details > > > https://wiki.openstack.org/wiki/Eventlet-removal#Identifying_Common_Pattern > > > > > > ## Important Dates > > > > > > - October 21st: cross PTG session. > > > - October 23rd: cross PTG session (Q&A session and Open Discussion > > > session) > > > > > > ## Call to Action > > > > > > We need your involvement to accelerate the progress of this initiative! > > > If your project is affected by Eventlet, now is the time to start planning > > > your migration. Teams are encouraged to take immediate steps in assessing > > > how to deactivate Eventlet or to transition to alternative solutions. > > > > > > We defined key roles and responsibilities. Those will help us to lower > > > the cost of the migration for our community. Teams are invited to assign > > > them now. Being ready as soon as possible will allow you to better take > > > advantage of the coming PTG sessions. > > > > > > Please find more details about these roles and their short terms tasks: > > > https://wiki.openstack.org/wiki/Eventlet-removal#Roles_and_Responsibilities > > > > > > ## Final Thoughts > > > > > > This is a significant step forward for improving the stability, > > > scalability, and maintainability of our projects. With your continued > > > support, we’re confident that this migration will bring substantial > > > benefits to the OpenStack community. > > > > > > If there are any questions or if you need assistance with your project’s > > > transition away from Eventlet, please join the discussion on the > > > #eventlet-removal channels (OpenStack irc, RedHat slack, mailing list). > > > > > > Thank you for your dedication and hard work! > > > > > > -- > > > Hervé Beraud > > > Senior Software Engineer at Red Hat > > > irc: hberaud > > > https://github.com/4383/ > > > > > >

1 year, 2 months

openstack-discuss search results for query "#eventlet-removal"