openstack-discuss search results for query "#eventlet-removal"
openstack-discuss@lists.openstack.org- 186 messages
Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary
by Clay Gerrard
> - Swift: Evaluating alternatives with impressive performance results
(FastWsgi showing 10x better performance!).
A swift core maintainer did a "hello world" application benchmark
comparison of eventlet & fast-wsgi and reported something similar:
FastWSGI server is about ~10 times faster than Eventlet WSGI server
AFAIK it's non-trivial and unproven if anything like that comparison would
hold up in a relatively complex application that's actually making backend
requests to other http/rest APIs or talking to ancillary services like
memcache and auth systems inline w/ every request/response.
IIUC, FastWSGI is using "one thread per request" (actually re-reading the
code that's no longer obvious to me, it's using libuv to handle connections
and parse sockets - but are all call_wsgi_application just blocking!?).
IMHO on modern systems a WSGI server using "one thread per request" is
probably reasonable for the kind of concurrency a swift proxy server
running CPU intensive operations like encryption and erasure coding (and
even decoding JWTs) will want to handle (i.e scale wide, more proxies doing
less concurrent RPS, e.g. with 100s of proxy-application worker nodes we
turn down max_clients from default 1000 to closer to 100 as part of a
solution to provide some back-pressure against overly aggressive concurrent
request spikes), and on the storage nodes OTOH one-thred-per-request would
probably *help* provide more *consistent* performance in the face of full
disks hitting seeks and flush when their blocking filesystem and sqlite
database calls are starving the hub anyway.
The "hardest" part of a swift migration off eventlet from my perspective is
the gratuitous use of lightweight concurrency to talk to multiple backend
services *within a request* - i.e. each request greenlet spawns/waits on
*multiple* greenlets (think trio nursery) like "connect to 12 backend
servers and stream each EC chunk for every client segment concurrently" or
"start a metadata server update request while waiting on a thread doing an
fsync and if either errors add a entry to a repair journal before
responding" using custom wrappers around GreenPool like a GreenAsyncPile
https://github.com/NVIDIA/swift/blob/master/swift/common/utils/__init__.py#…
Can a FastWSGI one-thread-per-request server still share a
ThreadPoolExecutor if you want to offload waiting on a blocking call while
running other coroutines talking to non-blocking APIs like a socket w/i
each request thread? I don't think I see why not...
Honestly the idea of a focused effort to wean a large system like swift off
of eventlet is daunting to say the least; certainly difficult to do
"optionally" or "experimentally" - it seems like once we port some small
backend service like ... idk the account-auditor - even "just" to use some
kind asyncio wrapper ... I think you just want to *delete* any kind of
"optional" monkey patching? Or is there some way towards a magic
translator (awaitlet?) that lets us try turning off "transparent single
thread w/ implicit concurrency" for "experimental multiple threads each
with explicit cooperative concurrency"? IIUC at some point near the bottom
of the layers of translation wrappers (to help avoid plumbing "async def"
and "return await" all the way from your "__main__ asyncio.run" to the very
bottom of your app) - you DO *eventually* get to the bottom and have to
s/from eventlet.green.http.client import HTTPConnection/from some_async_lib
import AwaitableHTTPConnectionLike/ *right*??? LIke a minimum you have to
"__main__ s/monkey_patch/asyncio.run/" - then you get to play games where
you can say "return awaitlet" instead of every function being a
generator/coroutine ... but whereas "chunk = sock.read()" would "just"
block the greenlet you'd spawned while "doing other stuff in that same
request" we do now actually have to write *some* async def coroutines? Or
maybe at the top you "if BETTER_CONCURRENCY: asyncio.run else monkey_patch"
and at the bottom do "return AwaitableConnection if BETTER_CONCURRENCY else
AwaitableLookingButActuallySomehowJustBoringBustedGreenTransparentBlockingConnection"???
I'd love to do some targeted experiments to get a feel for the options; I'm
not sure where to start!
On Thu, Apr 17, 2025 at 10:44 AM Herve Beraud <hberaud(a)redhat.com> wrote:
> Following the intensive discussions at the April 2025 Project Teams
> Gathering about the Eventlet removal initiative, this thread summarizes the
> current status, challenges, and strategies of all the OpenStack teams with
> a transversal perspective.
>
> A full and detailed report is available at:
> https://removal.eventlet.org/guide/openstack/flamingo-ptg/
>
> ## Python 3.13 Compatibility
>
> The countdown has begun! As documented in the April 2025 PTG discussions,
> the OpenStack community is accelerating efforts to remove Eventlet
> dependencies across all projects. This initiative has become critical due
> to Eventlet's compatibility problems with Python 3.13 and the upcoming
> "GILectomy" (PEP 703).
>
> With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing a
> hard deadline for this work
>
> https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461
>
> ## Migration Status
>
> ### Significant Progress
> - Octavia: Fully migrated since 2017! Their approach is now documented as
> a community case study.
> - Mistral: Almost there with a comprehensive migration approach.
> - Neutron: Significant progress with numerous patches already merged.
> - Glance: Can be deployed without Eventlet, though some optional features
> still need work.
>
> ### Work in Progress
> - Nova: Planning a service-by-service migration with dual-mode support
> during transition.
> - Swift: Evaluating alternatives with impressive performance results
> (FastWsgi showing 10x better performance!).
> - Manila, Cinder, Heat: All have active plans for the Flamingo cycle.
> - Designate, Blazar, Watcher: Starting their migration journeys.
> - Ironic: Facing complex challenges, particularly with the Ironic Python
> Agent component.
>
> ## Technical Spotlight: Oslo.service's Threading Backend
>
> A key development is the new threading backend for oslo.service that
> eliminates the Eventlet dependency:
> - Review in progress:
> https://review.opendev.org/c/openstack/oslo.service/+/945720
> - No longer provides WSGI support
> - Each service will need to deprecate implementations dependent on
> Eventlet's WSGI server
>
> ## Migration Strategies: One Size Doesn't Fit All
>
> Projects are adopting diverse approaches:
>
> - Dual-mode support: Nova and Glance are supporting both Eventlet and
> native threads during transition
> - Canary approach: Swift is considering starting with proxy nodes
> - Component-by-component: Cinder is starting with the Volume Manager
> - Complete replacement: Heat is planning a full discontinuation of WSGI
> server implementations
>
> ## Nova's Detailed Roadmap
>
> Nova has a comprehensive two-cycle plan:
>
> Flamingo Cycle (2025.2):
> - API modernization
> - Architecture updates with environment variable controls
> - Performance improvements
> - Test transition to native threading
>
> Guppy Cycle (2026.1):
> - Core event loop conversion
> - Adoption of oslo.service's new threading backend
>
> ## Issues to Watch: RabbitMQ Heartbeat Problems
>
> The problem with RabbitMQ heartbeats story:
> - Timeouts and API failures in "green" environments
> - Partial solution using pthreads exists but has logging issues
> - Track the fix: https://review.opendev.org/c/openstack/oslo.log/+/937729
>
> ## How to Get Involved
>
> - Join the #openstack-eventlet-removal channel on OFTC
> - Review the official guide: https://removal.eventlet.org/
> - Look for patches under the "eventlet-removal" topic:
> https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22
>
> ## Recommended Reading
>
> For teams starting their migration:
> 1. Official goal documentation:
> https://governance.openstack.org/tc/goals/selected/remove-eventlet.html
> 2. Migration preparation guide:
> https://removal.eventlet.org/guide/preparing-for-migration/
> 3. Octavia case study:
> https://removal.eventlet.org/guide/case-studies/octavia/
>
> ## Stay Updated
>
> - The complete version of this report is available online at:
> https://removal.eventlet.org/guide/openstack/flamingo-ptg/
> - Track the progress of the migration across all projects at:
> https://removal.eventlet.org/guide/openstack/#migration-status
>
> Thanks for reading!
>
> PS: This PTG summary is based on the etherpads following the April 2025
> PTG discussions (https://ptg.opendev.org/etherpads.html) For more
> details, refer to the full report (
> https://removal.eventlet.org/guide/openstack/flamingo-ptg/)
>
--
Clay Gerrard
210 788 9431
8 months, 2 weeks
[watcher] 2nd call for maintainers - MAAS integration
by Douglas Viroel
Hi all,
The Watcher team had a session[1] during the last PTG about the impacts of
service integrations that have lacked maintainers, documentation, and/or
testing coverage in CI. The MAAS (Metal As A Service) integration still
lacks maintainers and is impacting the eventlet removal effort within
Watcher, since it contains eventlet-dependent code that requires
maintenance and validation.
In the 2025.2 development cycle, we had a first call for maintainers[2] and
we didn't receive any feedback.
This is a second call for maintainers interested in supporting MAAS in
Watcher, to help fill these gaps (eventlet removal, CI testing,
documentation).
The Watcher team plans to *deprecate* this integration during the 2026.1
cycle if we don't find anyone to assist in maintaining it.
[1] https://etherpad.opendev.org/p/watcher-2026.1-ptg#L177
[2]
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
Thanks and regards,
--
Douglas Viroel - dviroel
1 month, 2 weeks
#eventlet-removal - Progress & Updates - Flamingo Past Milestone 3
by Herve Beraud
TL;DR: We just recently passed Flamingo's milestone 3 and since the last ML
update we observed tremendous progress. Ironic, Barbican, Heat completed
their migration. Futurist has been improved with new useful features for
threading mode. The migration timeline has been adjusted to match Nova's
dual mode support. PTG is under preparation. Statistics show good progress.
Eventlet now supports Python 3.14.
## OpenStack Updates
From a statistical perspective:
- almost 17% of the occurences of the runtime code initially impacted by
Eventlet have been tackled — outside of code dedicated to test, doc,
releasenotes, etc;
- if we include all the spectrum of occurrences to migrate — code, tests,
doc, examples — it represent ~14% of the entire code base already migrated;
- more than 40% of all impacted repositories have made some progress in
link with the migration;
- 13 projects have been fully migrated since the beginning;
- more than 332 patches related to the eventlet migration have been
submitted to gerrit.
From the teams perspective, we can highlight
Ironic completed its migration, for the occasion wrote an excellent
write-up [1] — thanks Julia — I strongly encourage you to read it.
Barbican [2] and heat [3] successfully completed their migrations too. They
are now fully threading based.
Dmitry in collaboration with the Ironic team implemented a
DynamicPoolExecutor in Futurist [4]. It helps replace dynamic GreenPool.
Oslo.service's backend system is now documented [5].
There is still ongoing work on several OpenStack deliverables: Neutron,
Nova, Manila, Magnum, Mistral, Watcher, Designate, Trove, etc. All these
teams demonstrated awesome progress.
The migration timeline has been adjusted at the governance level to match
Nova's dual mode support [6].
There is ongoing preparation of the PTG sessions. I think there will be a
short slot during one of the TC sessions, and surely a fully dedicated one
hour slot to host cross team open discussions.
## Eventlet Updates
The support of Python 3.14 has been introduced within Eventlet 0.40.2.
A vulnerability (CVE-2025-58068, CVSS 6.3) in Eventlet’s WSGI parser
allowing HTTP Request Smuggling (bypass controls, cache poisoning, user
targeting) has been found. It has been fixed within Eventlet 0.40.3. A
security advisory has been published [7].
## Call to Action
That would be worth considering introducing small experiments as
opportunities to identify or learn any weaknesses in new threading
architectural models. That could start by preparing and performing some
perf scale testing at your deliverables levels. That will help you to
define the right scaling configuration for your deliverables. You can draw
inspiration from Neutron's rally jobs [8] and from Ironic's scripting
created for benchmark purposes [9]. More than a year ago, Dan sparked the
performance testing topic idea [10], that's the right time to consider it
more deeply.
If you do not start the migration of your deliverables, I'd encourage you
to plan and prioritize it as soon as possible. Time flies and the later
your migration starts the harder it will be. Python 3.15 comes with its
share of breaking changes [11] so I'd not suggest waiting for that time.
If you are scheduling a PTG session on your side about Eventlet, then
please let us know by updating the eventlet removal wiki page [12].
If your service is now migrated, I'd invite you to tag the launchpad bugs
which were related to Eventlet, and, if it makes sense to close them. You
are now entering a clean phase.
## Final Thoughts
During Flamingo we observed tremendous progress, and we are now going
straight toward the final release of Flamingo.
The PTG, and Gazpacho that will be the logical continuation of the jobs
already done.
Flamingo will mark a significant release due to the fully migrated bricks
and services. They will allow us to enter a beneficial feedback loop.
The migration path is now well cleared by all the previous actions and
changes applied into OpenStack.
Postponing the beginning of the migration will make your life harder. We
want to prevent this from happening to you.
Your journey start here https://removal.eventlet.org/
If you need information join-us on the #openstack-eventlet-removal OFTC
channel.
Thanks to all the people who made this migration possible until now.
Thanks for your attention.
[1] https://ironicbaremetal.org/blog/coming-soon-threading/
[2] https://review.opendev.org/c/openstack/barbican/+/955301
[3] https://review.opendev.org/c/openstack/heat/+/952651
[4]
https://docs.openstack.org/futurist/latest/reference/index.html#futurist.Dy…
[5] https://docs.openstack.org/oslo.service/latest/user/backend.html
[6] https://review.opendev.org/c/openstack/governance/+/952903
[7] https://github.com/advisories/GHSA-hw6f-rjfj-j7j7
[8] https://opendev.org/openstack/neutron/src/branch/master/rally-jobs
[9] https://opendev.org/openstack/ironic/src/branch/master/tools/benchmark
[10]
https://review.opendev.org/c/openstack/governance/+/902585/comment/979db3f6…
[11] https://docs.python.org/es/dev/whatsnew/3.15.html#threading
[12]
https://wiki.openstack.org/wiki/Eventlet-removal#Project-specific_events
--
Hervé Beraud
Principal Software Engineer at Red Hat
irc: hberaud
https://github.com/4383/
3 months, 4 weeks
Re: #eventlet-removal - OFTC IRC meeting time slot poll - Express your choice!
by Herve Beraud
The selected time slot is Tuesday 3pm UTC, each first and third Tuesday of
the month.
https://framadate.org/rUghO3j6ZkRhKD7m
Next meeting is next week (Nov 19), here is the list of our meetings for
epoxy:
https://etherpad.opendev.org/p/epoxy-eventlet-tracking
All coming dates are available here
https://wiki.openstack.org/wiki/Eventlet-removal#Save_the_Date
Thanks for your time
Le mer. 6 nov. 2024 à 13:00, Herve Beraud <hberaud(a)redhat.com> a écrit :
> Voting will end the day after tomorrow. Many people already expressed
> their voice, and you?
> Last chance to pick your prefered time slot:
> https://framadate.org/rUghO3j6ZkRhKD7m
>
> Le mer. 30 oct. 2024 à 11:23, Herve Beraud <hberaud(a)redhat.com> a écrit :
>
>> Hey Stackers,
>>
>> We propose to schedule a one hour OFTC IRC meeting
>> (#opentack-eventlet-removal) each first and third week of each month.
>>
>> Let's find together the right UTC time slot for our one hour OFTC IRC
>> meeting. Once the vote is done, meetings will be scheduled each first and
>> third <selected time slot in the poll> of each month.
>>
>> Please, vote here: https://framadate.org/rUghO3j6ZkRhKD7m
>>
>> Votes will be considered as closed by Nov. 8th
>>
>> First meeting will be forced to 4pm UTC on Tuesday Nov 5th and then moved
>> to the selected time slot for all the following meetings. See the proposed
>> time slots in the poll link above.
>>
>> Join us on the #opentack-eventlet-removal OFTC IRC channel.
>>
>> Thanks in advance for your votes
>> --
>> Hervé Beraud
>> Senior Software Engineer at Red Hat
>> irc: hberaud
>> https://github.com/4383/
>>
>>
>
> --
> Hervé Beraud
> Senior Software Engineer at Red Hat
> irc: hberaud
> https://github.com/4383/
>
>
--
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud
https://github.com/4383/
1 year, 1 month
Re: #eventlet-removal - Progress & Updates - Flamingo milestone 1
by Rodolfo Alonso Hernandez
Hello all:
This is the Neutron etherpad:
https://etherpad.opendev.org/p/neutron-eventlet-deprecation (we used
"deprecation" because we didn't have the tag "eventlet-removal" when we
created it)
So far, we have achieved the following tasks:
* Remove the usage of eventlet in the Neutron API (not the maintenance
method, periodic or RPC). Right now, the API only works with WSGI module.
* MIgrate the following agents:
** OVN agent
** OVN metadata agent
** Metadata agent
** SRIOV agent
The OVS agent, L3 agent and DHCP are the next ones to be migrated and are
very close. We are actually expecting the oslo.service new backend
("threading") to continue with these agents removal.
Regards.
On Wed, May 21, 2025 at 10:20 AM Herve Beraud <hberaud(a)redhat.com> wrote:
> Thanks Douglas.
> I also added your etherpad into the teams efforts tracker
> https://wiki.openstack.org/wiki/Eventlet-removal#Teams_efforts
>
> Le mar. 20 mai 2025 à 21:02, Douglas Viroel <viroel(a)gmail.com> a écrit :
>
>> Hi hberaud o/
>>
>> Thanks for the updates.
>> The Watcher team also started investigating eventlet usage in the code
>> and proposed some initial actions.
>> We started to follow up this effort in our weekly meeting too.
>> Our etherpad is the following:
>> https://etherpad.opendev.org/p/watcher-eventlet-removal
>>
>> BR,
>>
>> On Tue, May 20, 2025 at 3:51 PM Herve Beraud <hberaud(a)redhat.com> wrote:
>>
>>>
>>>
>>> Le mar. 20 mai 2025 à 18:47, Dmitry Tantsur <dtantsur(a)protonmail.com> a
>>> écrit :
>>>
>>>> Hi!
>>>>
>>>> Thank you for the update, here is the etherpad I created for Ironic:
>>>> https://etherpad.opendev.org/p/ironic-eventlet-removal
>>>>
>>>>
>>> Thank you Dimitry, that looks pretty amazing, kudos! Let me add it to
>>> our collection of efforts.
>>>
>>>
>>>> Dmitry
>>>>
>>>> On 5/20/25 6:44 PM, Herve Beraud wrote:
>>>> > TL;DR: We just passed the milestone 1 of Flamingo and at this point
>>>> big
>>>> > fishes like Neutron, Nova, Glance, and Mistral demonstrated awesome
>>>> > progress in their migration. Oslo.service's new backend is now usable
>>>> in
>>>> > real life use cases. Eventlet still suffers from a new version of
>>>> > Python. If you have not yet started your migration yet, then, please
>>>> > consider starting ASAP before the aircraft crash.
>>>> >
>>>> > ## OpenStack Updates
>>>> >
>>>> > Neutron's L3 agent is well on its way to running its WSGI features
>>>> > without using Eventlet and activating the new oslo.service threading
>>>> > backend at runtime. Allowing to decouple a bit more the needs of
>>>> > Eventlet at runtime.
>>>> >
>>>> > Nova scheduler is now able to start in threading mode without
>>>> requiring
>>>> > any usage of Eventlet at runtime. This was possible by also shifting
>>>> > nova-scheduler to the threading backend of oslo.service. More details
>>>> > are available here https://gibizer.github.io/categories/eventlet/
>>>> > <https://gibizer.github.io/categories/eventlet/>
>>>> >
>>>> > Activating the new threading backend of oslo.service required
>>>> devstack
>>>> > adaptation to allow having jobs running this execution mode. https://
>>>> > review.opendev.org/c/openstack/devstack/+/948436 <https://
>>>> > review.opendev.org/c/openstack/devstack/+/948436>. Everybody will be
>>>> > able to benefit from these adaptations.
>>>> >
>>>> > Glance continues its adaptation journey by migrating its functional
>>>> jobs
>>>> > to reduce its needs of Eventlet and to allow it to run jobs in
>>>> threading
>>>> > mode.
>>>> >
>>>> > oslo.service's new backend is on the way to be landed soon, and is
>>>> > waiting for additional reviews.
>>>> https://review.opendev.org/c/openstack/
>>>> > oslo.service/+/945720 <https://review.opendev.org/c/openstack/
>>>> > oslo.service/+/945720>
>>>> >
>>>> > ## Eventlet Updates
>>>> >
>>>> > Our CI was failing for almost 2 months, we were able to unlock this
>>>> > failing CI and hence we were able to merge several blocked reviews.
>>>> >
>>>> > Last week Eventlet 0.40.0 was released. This new version provides
>>>> fixes
>>>> > around OpenSSL 3.5+.
>>>> > The support of Python 3.8 and of PyPy were removed.
>>>> >
>>>> > The Python 3.13 issues are still not solved and require further work.
>>>> >
>>>> > ## Call to Action
>>>> >
>>>> > Flamingo is a crucial series in the realization of global migration
>>>> of
>>>> > OpenStack. We need your involvement to accelerate the progress of
>>>> this
>>>> > initiative!
>>>> >
>>>> > If your project is affected by Eventlet, now is the time to start
>>>> your
>>>> > migration. Teams are invited to read the migration guide available at
>>>> > https://removal.eventlet.org/ <https://removal.eventlet.org/> and
>>>> are
>>>> > encouraged to join the #openstack-eventlet-removal IRC channel to get
>>>> help.
>>>> >
>>>> > You'll benefit quickly from the advantages of running your
>>>> deliverables
>>>> > without using Eventlet.
>>>> >
>>>> > ## Final Thoughts
>>>> >
>>>> > Postponing the beginning of the migration will put additional
>>>> pressure
>>>> > related to growing runtime failures of Eventlet. We want to prevent
>>>> this
>>>> > from happening to you.
>>>> >
>>>> > We are now going straight to milestone 2 meaning that we will soon be
>>>> in
>>>> > the middle of Flamingo.
>>>> >
>>>> > The migration path is now well cleared by all the previous actions
>>>> and
>>>> > changes applied into OpenStack. Do not be afraid by the migration and
>>>> > take advantage of the collected experience.
>>>> >
>>>> > Your journey start here https://removal.eventlet.org/ <https://
>>>> > removal.eventlet.org/>
>>>> >
>>>> > Thanks for reading.
>>>> >
>>>> > --
>>>> > Hervé Beraud
>>>> > Principal Software Engineer at Red Hat
>>>> > irc: hberaud
>>>> > https://github.com/4383/ <https://github.com/4383/>
>>>> >
>>>>
>>>>
>>>>
>>>
>>> --
>>> Hervé Beraud
>>> Principal Software Engineer at Red Hat
>>> irc: hberaud
>>> https://github.com/4383/
>>>
>>>
>>
>> --
>> Douglas Viroel - dviroel
>>
>
>
> --
> Hervé Beraud
> Principal Software Engineer at Red Hat
> irc: hberaud
> https://github.com/4383/
>
>
7 months, 1 week
Re: Upstream Oslo IRC Meetings Are Back.
by Takashi Kajinami
I noticed that eventlet removal meeting is independent from
the oslo meeting proposed in this thread. I'm ok to have dedicated
slot to discuss oslo specific topics but may need to have clear scope
definition. (Since it was mentioned that eventlet removal is the core
topic we want to discuss in this meeting, right ?).
Regarding the time slot, EMEA/NA slots might work better for me because
I can join these after I settle down my own things, but I think it's
generally got to have more Asia friendly slots if anyone from Asia
(especially east-Asia) is interested in joining us.
On 10/31/24 7:07 PM, dbengt(a)redhat.com wrote:
>
>
> On 10/30/24 5:57 PM, Ghanshyam Mann <gmann(a)ghanshyammann.com> wrote:
>> ++ on alternate TZ meetings, and that way, EU and Asia TZ can be covered in one of the meetings.
>> To keep it simple, we can have separate doodle poll for each TZ meeting.
> Ok so maybe instead we can have one meeting per week. As you proposed like one for the Europeean and Asia and one for the us? I have create a first UTC pool here[1] afternoon and here[2] you have the morning one.
>
> [1] https://www.vyte.in/e/1cnp51e4sd [2] https://www.vyte.in/e/2j0to257cd
>
1 year, 2 months
[heat][ptg][ops] Flamingo PTG summary
by Takashi Kajinami
Hello,
Last week during the PTG the heat team gathered to discuss a few topics.
Thank you all who attended the session and the valuable discussions.
The details can be found in https://etherpad.opendev.org/p/apr2025-ptg-heat but
I'll leave some summary. There are a few plans about which we want to ask for
any feedback from operators so I'm adding [ops] tag here.
1. Eventlet removal
During the session we reviewed the current global plan to remove/replace
eventlet and discussed the plan to follow it in heat. Our current idea is
to remove/deprecate unmaintained code first so that we can shrink the scope.
At this moment we aim to deprecate and remove the following items.
- Standalone heat-api and heat-cfn-api
- These are deprecated due to deprecation of WSGI server implementation
from oslo.service, which depends on eventlet.
Users can run api server by external server mechanism such as mod_wsgi or uwsgi.
- heat-all service
- This is deprecated due to deprecation of WSGI server implementation
from oslo.service, which depends on eventlet. We expect this is not
really used in production
- legacy engine
- This was deprecated in favor of convergence. Note that removing this means
we no longer support adopt/abandon/check feature.
Question to operators:
Does anyone have concern about deprecating/removing these items ?
Please be aware that the current heat team is quite small and we need
actual help to replace eventlet for these features.
2. CFN API
We discussed the current status of heat-cfntool which heavily depends on unmaintained
boto library. However we also agreed that replacing boto by boto3 requires some effort
due to the fact that boto3 does not allow overriding endpoint quite easily.
While we aim to deprecate CFN api, we agreed that we still have to asses the impact
because CFN API is used for signaling by default.
Question to operators:
Is anyone using CFN API in any other use case than signaling ?
For the first topic, if we don't hear any objections, we'll follow the plan to
move the removal work forward.
Thank you,
Takashi
--
Takashi Kajinami
irc: tkajinam
github: https://github.com/kajinamit
launchpad: https://launchpad.net/~kajinamit
8 months, 2 weeks
Re: #eventlet-removal [ptg] 2025.2 Flamingo PTG summary
by Thomas Goirand
Hi,
On 4/17/25 17:44, Herve Beraud wrote:
> With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing
> a hard deadline for this work
>
> https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461
If I may: Debian 13 (aka: Trixie) is already in (soft) freeze, and
shipping Python 3.13 only also. Harder freeze is in a month, complete
freeze is still not announced yet. I'm also very worried about the state
of Eventlet with Epoxy on Trixie (I haven't had time to tempest test it
yet).
Is there a new eventlet release that's going to fix more issues? Will
you continue working on the patch you pointed at, and make sure
OpenStack continues to work with it?
Thanks for all your work on this,
Cheers,
Thomas Goirand (zigo)
8 months, 2 weeks
[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?
by Stephen Finucane
o/
tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from
DevStack. I'd also like to remove the ability to deploy services under eventlet,
where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the
service in question, there are currently 3 possible ways an API can be deployed
by DevStack:
* Eventlet server
* Apache + uWSGI
* Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set
to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES
* WSGI_MODE
* Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and
most obviously, we are working on eventlet removal, which will necessarily
involve eventlet server removal [1]. In addition, there is a goal [2][3] to
remove use of '.wsgi' scripts in favour of Python module paths. Quite a few
services have migrated to this already, and those that haven't are going to need
to do so by time pip 25.0 comes out (since that removes the deprecated legacy
editable install flow [4] meaning these files will not be generated by pbr and
DevStack deployments will start failing...you have been warned). However,
mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in
DevStack still before last week, so to complete the wsgi script -> module
migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're
going to have to either (a) package a .wsgi script in DevStack or (b) remove the
ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths
immediately and deprecate the config options for removal to force DevStack
plugins to adapt. However, some services still want to test eventlet server for
their API while they stabilise support for standard WSGI-based deployments
(neutron), while others explicitly rely on evenlet to generate a single
standalone binary (ironic, glance, ...). That means we can't just go remove
everything now and must instead take a phased approach. I would therefore like
to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and
deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can
continue to use mod_wsgi but DevStack won't.
* Entirely remove the ability to deploy under eventlet from services that are
not currently testing this. This includes Nova and Cinder.
* Invert the default for services that are testing eventlet, so that they test
with uWSGI by default now and jobs will need to opt-in to eventlet rather
than opt-in to uWSGI. I think this just affects Neutron.
* Optionally, and likely only one Neutron is no longer testing eventlet (next
cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI
config options entirely and replace it with {SERVICE}_STANDALONE options like
Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the
things that I am less clear about, such as Swift's readiness for deploying under
uWSGI. I would be grateful if folks from the individual service teams could take
a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
[2] https://review.opendev.org/c/openstack/governance/+/902807
[3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…
[4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
1 year, 2 months
Re: #eventlet-removal - Progress & Update
by smooney@redhat.com
On Wed, 2024-10-09 at 12:01 +0200, Sylvain Bauza wrote:
> Le ven. 4 oct. 2024 à 17:41, engineer2024 <engineerlinux2024(a)gmail.com> a
> écrit :
>
> > That's encouraging news. I hope the new alternative would support a python
> > debugger like pdb. This will help the new users to understand the
> > flow better. So when can we expect initial release atleast beta of eventlet
> > free nova components ?
> >
> >
> That's an excellent question that I don't have an answer. We started to
> work on it on Dalmatian but for the moment, I don't know whether the owner
> could continue to work on it on Epoxy or if someone else would do it.
i dont think we will ever have a "beta" release without eventlet
our expection is that we will do a phased removal.
so the iniall goal was to to tackel some of the low hanging furit in dalmation
the first part of that was spliting the delivberable in nova that actully need eventlet
vs those that dont
i.e. the command line clinets were monkey patched but never needed eventlet so those
were simple to seprate to remove the monkey patching
thats what https://review.opendev.org/c/openstack/nova/+/904424 does
the next step is to remove our use of eventlets thread pool and replace it with futureist's
version.
https://review.opendev.org/c/openstack/nova/+/922497/5
https://review.opendev.org/c/openstack/nova/+/905287/9
https://review.opendev.org/c/openstack/nova/+/917962/8
the third step is to remove the only usage of eventlet in the nova-api wsgi module
https://review.opendev.org/c/openstack/nova/+/905284/7
i may or may not have time to work on this in 2025.1
i will likely try to make some progress but i may need to hand this off to someone
else or rope in some other to help. if we can compelte what i had orginally
scoped in https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 for 2025.1
that woudl be a good first step.
next steps would be two fold.
first we need to change how spawn core service event loop by using the new backend being
added by https://review.opendev.org/c/openstack/oslo-specs/+/927503
second we need to change how we handel rpc requests.
when the agent recives an rpc instest of executiong the function on the main thread we
need to dispatch that to either a GreenThreadPoolExecutor when we are monkey patched (using eventlet)
https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L318
or a ThreadPoolExecutor https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L99
when we are not usign eventlet.
that is requried so we can still have concurrent interleaved rpc processing and allow the
threaded version to just block waiting on future as needed.
that bit i have not figured out exactly where to do that yet.
but factoring out the rpc handling to use the executor interface would likely be eventlet-removal-part-2 (2025.2)
since that can be done without a dep onthe oslo.service change and then
eventlet-removal-part-3 2026.1 woudl be converting to the new oslo.servce backend and any remaining clean up.
that is optimistic and a lot of work so it may take longer or we might be able to start some other things earlier.
tehre is a lot of good work in https://wiki.openstack.org/wiki/Eventlet-removal#Migration_Plan also.
nova at present is not planning to use asynio but that does nto mean we never will
we just dont have a high enough concurrence requireemtn to need it so trhead pools
shoud be sufficient
once we are using them there is no reaosn we coudl not have an AsyncioPoolExecutor
using awaitlet or explore using asyncio more directly
to your debugger point any servie that is not monkey patched will gain the ablity to just run
in a debugger normally i.e. once we get rid of using it in the api wsgi module you should be able to just
debug it but keep in mind that nova is a distributed system os if you set a breakpoint you will very likly
cause request to time out. debuging a distribute system is not the same as debuging a application running on
a single host and never will be.
that is not a goal of this work its just a side benifit.
>
> -Sylvain
>
> On Fri, 4 Oct 2024, 20:38 Herve Beraud, <hberaud(a)redhat.com> wrote:
> >
> > > We’re excited to share the latest updates on our ongoing efforts to phase
> > > out Eventlet across our various OpenStack projects. Below you’ll find a
> > > summary of our progress, key milestones, and important dates to keep in
> > > mind as we move forward with this critical initiative.
> > >
> > > ## Current Status
> > >
> > > We’ve completed the initial analysis phase across more than 120 OpenStack
> > > projects. This includes identifying the various patterns where Eventlet is
> > > currently in use and evaluating where it can be safely removed or disabled.
> > > Here are some highlights:
> > >
> > > - Completed Analysis: Projects where Eventlet is used include (but are
> > > not limited to): Aodh, Barbican, Nova, Neutron, Tacker, Taskflow, Trove,
> > > Venus, Watcher, and Zun.
> > >
> > > - Eventlet Optional: Several projects have been identified where Eventlet
> > > seems to be optional and can be disabled with minimal impact. For these
> > > projects, we will move forward the discussion with teams to try
> > > deactivating Eventlet in upcoming updates. Once done, projects of this kind
> > > would be able to start their migration with more serenity.
> > >
> > > - Eventlet Required: Some projects still rely heavily on Eventlet for
> > > core functionality. For these, we are going to develop specific migration
> > > plans.
> > >
> > > ## Migration Plans in Progress
> > >
> > > - oslo.db: Preparing its Asyncio based engine facade;
> > > - oslo.service: Refactoring plans being discussed to minimize disruption
> > > during the migration;
> > > - glance: Refactoring code to use native thread model rather than Eventlet
> > >
> > > More details can be found here:
> > > https://review.opendev.org/q/topic:%22eventlet-removal%22
> > >
> > > ## Latest Updates
> > >
> > > - Community Goal Transition: The ongoing community goal for removing
> > > Eventlet is now in the process of becoming an official "selected" goal,
> > > https://review.opendev.org/c/openstack/governance/+/931254.
> > >
> > > - Pattern Identification Completed: We’ve identified key patterns related
> > > to Eventlet usage, including its common integration with monkey_patch(),
> > > eventlet.Timeout, eventlet.GreenPool, and eventlet.wsgi.server(). Visit the
> > > official wiki to get more details
> > > https://wiki.openstack.org/wiki/Eventlet-removal#Identifying_Common_Pattern
> > >
> > > ## Important Dates
> > >
> > > - October 21st: cross PTG session.
> > > - October 23rd: cross PTG session (Q&A session and Open Discussion
> > > session)
> > >
> > > ## Call to Action
> > >
> > > We need your involvement to accelerate the progress of this initiative!
> > > If your project is affected by Eventlet, now is the time to start planning
> > > your migration. Teams are encouraged to take immediate steps in assessing
> > > how to deactivate Eventlet or to transition to alternative solutions.
> > >
> > > We defined key roles and responsibilities. Those will help us to lower
> > > the cost of the migration for our community. Teams are invited to assign
> > > them now. Being ready as soon as possible will allow you to better take
> > > advantage of the coming PTG sessions.
> > >
> > > Please find more details about these roles and their short terms tasks:
> > > https://wiki.openstack.org/wiki/Eventlet-removal#Roles_and_Responsibilities
> > >
> > > ## Final Thoughts
> > >
> > > This is a significant step forward for improving the stability,
> > > scalability, and maintainability of our projects. With your continued
> > > support, we’re confident that this migration will bring substantial
> > > benefits to the OpenStack community.
> > >
> > > If there are any questions or if you need assistance with your project’s
> > > transition away from Eventlet, please join the discussion on the
> > > #eventlet-removal channels (OpenStack irc, RedHat slack, mailing list).
> > >
> > > Thank you for your dedication and hard work!
> > >
> > > --
> > > Hervé Beraud
> > > Senior Software Engineer at Red Hat
> > > irc: hberaud
> > > https://github.com/4383/
> > >
> > >
1 year, 2 months