Search results for "#eventlet-removal" - openstack-discuss

[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?

by Stephen Finucane

o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/

9 months, 4 weeks

Re: #eventlet-removal - Progress & Update

by smooney＠redhat.com

On Wed, 2024-10-09 at 12:01 +0200, Sylvain Bauza wrote: > Le ven. 4 oct. 2024 à 17:41, engineer2024 <engineerlinux2024(a)gmail.com> a > écrit : > > > That's encouraging news. I hope the new alternative would support a python > > debugger like pdb. This will help the new users to understand the > > flow better. So when can we expect initial release atleast beta of eventlet > > free nova components ? > > > > > That's an excellent question that I don't have an answer. We started to > work on it on Dalmatian but for the moment, I don't know whether the owner > could continue to work on it on Epoxy or if someone else would do it. i dont think we will ever have a "beta" release without eventlet our expection is that we will do a phased removal. so the iniall goal was to to tackel some of the low hanging furit in dalmation the first part of that was spliting the delivberable in nova that actully need eventlet vs those that dont i.e. the command line clinets were monkey patched but never needed eventlet so those were simple to seprate to remove the monkey patching thats what https://review.opendev.org/c/openstack/nova/+/904424 does the next step is to remove our use of eventlets thread pool and replace it with futureist's version. https://review.opendev.org/c/openstack/nova/+/922497/5 https://review.opendev.org/c/openstack/nova/+/905287/9 https://review.opendev.org/c/openstack/nova/+/917962/8 the third step is to remove the only usage of eventlet in the nova-api wsgi module https://review.opendev.org/c/openstack/nova/+/905284/7 i may or may not have time to work on this in 2025.1 i will likely try to make some progress but i may need to hand this off to someone else or rope in some other to help. if we can compelte what i had orginally scoped in https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 for 2025.1 that woudl be a good first step. next steps would be two fold. first we need to change how spawn core service event loop by using the new backend being added by https://review.opendev.org/c/openstack/oslo-specs/+/927503 second we need to change how we handel rpc requests. when the agent recives an rpc instest of executiong the function on the main thread we need to dispatch that to either a GreenThreadPoolExecutor when we are monkey patched (using eventlet) https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L318 or a ThreadPoolExecutor https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L99 when we are not usign eventlet. that is requried so we can still have concurrent interleaved rpc processing and allow the threaded version to just block waiting on future as needed. that bit i have not figured out exactly where to do that yet. but factoring out the rpc handling to use the executor interface would likely be eventlet-removal-part-2 (2025.2) since that can be done without a dep onthe oslo.service change and then eventlet-removal-part-3 2026.1 woudl be converting to the new oslo.servce backend and any remaining clean up. that is optimistic and a lot of work so it may take longer or we might be able to start some other things earlier. tehre is a lot of good work in https://wiki.openstack.org/wiki/Eventlet-removal#Migration_Plan also. nova at present is not planning to use asynio but that does nto mean we never will we just dont have a high enough concurrence requireemtn to need it so trhead pools shoud be sufficient once we are using them there is no reaosn we coudl not have an AsyncioPoolExecutor using awaitlet or explore using asyncio more directly to your debugger point any servie that is not monkey patched will gain the ablity to just run in a debugger normally i.e. once we get rid of using it in the api wsgi module you should be able to just debug it but keep in mind that nova is a distributed system os if you set a breakpoint you will very likly cause request to time out. debuging a distribute system is not the same as debuging a application running on a single host and never will be. that is not a goal of this work its just a side benifit. > > -Sylvain > > On Fri, 4 Oct 2024, 20:38 Herve Beraud, <hberaud(a)redhat.com> wrote: > > > > > We’re excited to share the latest updates on our ongoing efforts to phase > > > out Eventlet across our various OpenStack projects. Below you’ll find a > > > summary of our progress, key milestones, and important dates to keep in > > > mind as we move forward with this critical initiative. > > > > > > ## Current Status > > > > > > We’ve completed the initial analysis phase across more than 120 OpenStack > > > projects. This includes identifying the various patterns where Eventlet is > > > currently in use and evaluating where it can be safely removed or disabled. > > > Here are some highlights: > > > > > > - Completed Analysis: Projects where Eventlet is used include (but are > > > not limited to): Aodh, Barbican, Nova, Neutron, Tacker, Taskflow, Trove, > > > Venus, Watcher, and Zun. > > > > > > - Eventlet Optional: Several projects have been identified where Eventlet > > > seems to be optional and can be disabled with minimal impact. For these > > > projects, we will move forward the discussion with teams to try > > > deactivating Eventlet in upcoming updates. Once done, projects of this kind > > > would be able to start their migration with more serenity. > > > > > > - Eventlet Required: Some projects still rely heavily on Eventlet for > > > core functionality. For these, we are going to develop specific migration > > > plans. > > > > > > ## Migration Plans in Progress > > > > > > - oslo.db: Preparing its Asyncio based engine facade; > > > - oslo.service: Refactoring plans being discussed to minimize disruption > > > during the migration; > > > - glance: Refactoring code to use native thread model rather than Eventlet > > > > > > More details can be found here: > > > https://review.opendev.org/q/topic:%22eventlet-removal%22 > > > > > > ## Latest Updates > > > > > > - Community Goal Transition: The ongoing community goal for removing > > > Eventlet is now in the process of becoming an official "selected" goal, > > > https://review.opendev.org/c/openstack/governance/+/931254. > > > > > > - Pattern Identification Completed: We’ve identified key patterns related > > > to Eventlet usage, including its common integration with monkey_patch(), > > > eventlet.Timeout, eventlet.GreenPool, and eventlet.wsgi.server(). Visit the > > > official wiki to get more details > > > https://wiki.openstack.org/wiki/Eventlet-removal#Identifying_Common_Pattern > > > > > > ## Important Dates > > > > > > - October 21st: cross PTG session. > > > - October 23rd: cross PTG session (Q&A session and Open Discussion > > > session) > > > > > > ## Call to Action > > > > > > We need your involvement to accelerate the progress of this initiative! > > > If your project is affected by Eventlet, now is the time to start planning > > > your migration. Teams are encouraged to take immediate steps in assessing > > > how to deactivate Eventlet or to transition to alternative solutions. > > > > > > We defined key roles and responsibilities. Those will help us to lower > > > the cost of the migration for our community. Teams are invited to assign > > > them now. Being ready as soon as possible will allow you to better take > > > advantage of the coming PTG sessions. > > > > > > Please find more details about these roles and their short terms tasks: > > > https://wiki.openstack.org/wiki/Eventlet-removal#Roles_and_Responsibilities > > > > > > ## Final Thoughts > > > > > > This is a significant step forward for improving the stability, > > > scalability, and maintainability of our projects. With your continued > > > support, we’re confident that this migration will bring substantial > > > benefits to the OpenStack community. > > > > > > If there are any questions or if you need assistance with your project’s > > > transition away from Eventlet, please join the discussion on the > > > #eventlet-removal channels (OpenStack irc, RedHat slack, mailing list). > > > > > > Thank you for your dedication and hard work! > > > > > > -- > > > Hervé Beraud > > > Senior Software Engineer at Red Hat > > > irc: hberaud > > > https://github.com/4383/ > > > > > >

10 months

Re: #eventlet-removal - OFTC IRC meeting time slot poll - Express your choice!

by Herve Beraud

Voting will end the day after tomorrow. Many people already expressed their voice, and you? Last chance to pick your prefered time slot: https://framadate.org/rUghO3j6ZkRhKD7m Le mer. 30 oct. 2024 à 11:23, Herve Beraud <hberaud(a)redhat.com> a écrit : > Hey Stackers, > > We propose to schedule a one hour OFTC IRC meeting > (#opentack-eventlet-removal) each first and third week of each month. > > Let's find together the right UTC time slot for our one hour OFTC IRC > meeting. Once the vote is done, meetings will be scheduled each first and > third <selected time slot in the poll> of each month. > > Please, vote here: https://framadate.org/rUghO3j6ZkRhKD7m > > Votes will be considered as closed by Nov. 8th > > First meeting will be forced to 4pm UTC on Tuesday Nov 5th and then moved > to the selected time slot for all the following meetings. See the proposed > time slots in the poll link above. > > Join us on the #opentack-eventlet-removal OFTC IRC channel. > > Thanks in advance for your votes > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

9 months, 1 week

[eventlet-removal]When to drop eventlet support

by Balazs Gibizer

Hi Stackers! I would like to sync about the planned timeline of dropping eventlet support from OpenStack / Oslo. Nova definitely needs at least the full 2026.1 cycle to have a chance to transform the nova-compute service. But this plan already feels stretched based on the progress in the current cycle. So being conservative means we need the 2026.2 cycle as a buffer. Nova would like to keep a release where we support both eventlet and threading in parallel. So that operators can do the switching from eventlet to threading outside of the upgrade procedure. (This was an explicit request from them during the PTG). So 2026.2 could be that version where nova fully supports both concurrency mode, while eventlet can be marked deprecated. Then the 2027.1 release could be the first release dropping eventlet. However we need to align with the SLURP upgrade as well. 2026.1 is a SLURP. But in that release Nova might not be ready to have all services running in threading mode. So the 2026.1 - 2027.1 SLURP upgrade would force the operators to change the concurrency mode during the upgrade itself. I see two ways forward: * A) We say that operators who want to do the concurrency mode change outside of an upgrade could not skip the 2026.2 release, i.e. they cannot do SLURP directly from 2026.1. to 2027.1. * B) We keep supporting the eventlet mode in the 2027.1 release as well and only dropping support in 2028.1. What do you think? Cheers gibi

2 months

Re: [eventlet-removal][release][TC] When to drop eventlet support

by Sean Mooney

On 17/06/2025 18:52, Herve Beraud wrote: > I think there is an option C. > > Since the beginning of this initiative, the goal has always been to > phase out Eventlet cleanly from OpenStack components, particularly > Nova. The initial proposal targeted the 2027.2 release, at a time when > Nova had no plans to support a dual concurrency model. At that point, > Eventlet removal was not classified as a feature removal subject to > SLURP policy - it was considered an internal refactoring, with no > addition of transitional features requiring later removal. The idea > was to deprecate any Eventlet-related features, public APIs, and > configuration options as early as possible. As a result, SLURP policy > constraints were not a primary concern. > > But the situation has evolved. Nova has now committed to offering a > full release cycle (2027.1) where both concurrency models - Eventlet > and threading - are supported. This is not a technical convenience, > but a response to explicit operator requests: operators want the > ability to switch concurrency modes outside in case of problem during > at least one series. The full threading support in Nova will not be > ready before 2026.2, which is a non-SLURP release, and that the > dual-mode support must be preserved throughout 2027.1. unless this has changed since the ptg we were hoping for the first release to supprot runign without eventlet to be 2026.1 and the defautl to chagne in 2026.2. so 2026.1 would be the slrup release where operator woudl chage to eventlet before upgrading to 2027.1 > > However, current SLURP policy requires that a removal can only happen > in a SLURP release. That is incorrect. deprecation can only happen in slurps removals can happen in either provided the feature is deprecated in the prior slurp. > According to Nova's plan that excludes a removal in 2027.1. that is also incorrect removal is only exclded if we dont get to full threadign supprot in 2026.1 we have not deprecated evenlet supprot yet and we dont plan do do that formally until we have show that nova can work without it. > If we follow the rules strictly, Eventlet removal would be delayed to > 2028.1, adding several months of dual-mode support, increased > maintenance complexity, and growing technical debt. again this is very much misrepresenting the current plan. > > From a technical standpoint, it’s important to highlight that upcoming > Python versions (3.14 and 3.15) introduce major changes to the GIL > [1], threading behavior, and RLocks - all areas [2] that have > historically been problematic in Eventlet. We’ve already had to > implement significant patches for Python 3.13. There is also > uncertainty around which Python versions will be adopted by the > various Linux distributions and by our own supported runtime > environments, making long-term compatibility assumptions with Eventlet > increasingly risky, but in all the cases it will put additional > pressure on us. Continuing to support Eventlet until 2028 is > effectively betting on the stability of a module that is already on > life support, with very few contributors available to maintain it > beyond 2027 (see Jay's note about GR-OSS). More critically, there is > no guarantee today that Eventlet, in its current state, will remain > robust enough to reliably support a fallback path in case a regression > forces a rollback from threading to Eventlet. > > This is why I propose a pragmatic and balanced compromise: we maintain > the planned deprecation (we might have to adjust a little bit those > from oslo), but we make a documented and limited exception to allow > Eventlet removal in 2027.2, which is a non-SLURP cycle. This exception > would be coordinated with the Technical Committee and the release team > and is justified by: > > - the commitment made by Nova to offer a full cycle of dual-mode > support in 2027.1; again that was not what we commited too. we aim to provide that in 2026.1 if we dont make that teh deprecation fo eventlet supprot will slip to 2026.2 and we will supprot both in 2027.1 and do the removal in 2027.2. supprot for eventlet in 2027.1 is predicated in not completeing the work in 2026.1. > - the high technical cost of maintaining dual concurrency models; > - the fast evolution of Python and the increasing risk of > incompatibilities with Eventlet; > - the limited engineering resources available to sustain Eventlet > maintenance; > - and the need to align with operator expectations while still > securing the long-term health of the platform. > > This approach gives Nova the entire 2027.1 cycle to deliver a complete > and stable dual-mode implementation, as promised to operators. At the > same time, it enables the rest of the ecosystem to move forward > safely, without relying on a dependency that is increasingly brittle > and obsolete. > > In large open source projects, it is not uncommon for policies to be > temporarily adjusted in the face of structural or technical > emergencies. This is not a convenience exception - it’s a reasoned and > responsible deviation from the norm, driven by necessity. > > For this reason I add the release team and the TC to the party. Please > share your feedback with us. > > [1] > https://docs.python.org/es/dev/whatsnew/3.14.html#whatsnew314-free-threaded… > [2] https://docs.python.org/es/dev/whatsnew/3.15.html#threading. > > Le lun. 16 juin 2025 à 18:06, Sean Mooney <smooney(a)redhat.com> a écrit : > > > On 16/06/2025 15:33, Dmitriy Rabotyagov wrote: > > Also let's keep in mind, that only nova (with placement) will > spawn 64 > > threads on such setup by default. > > yep see my other reply. > > you mixign up workers and eventlet threads. > > those are two very diffent things. > > we likely shoudl change the upstream default for workers to 1. > > we do tend to override that by defualt in installer tools. > > > > > And then all really depends on set of services to launch on such > setup. > > > > So from deployment tooling protective you have all required data to > > rollout not instantly oom-ing setup at cost of amount of request > > services can process in parallel. > > > > On Mon, 16 Jun 2025, 15:24 Dmitriy Rabotyagov, > > <noonedeadpunk(a)gmail.com> wrote: > > > > In case you try to use a 32gb box with 16 cores as a controller > > for OpenStack - it will blow off with default amount of workers > > for wsgi and /or eventlet apps. > > > > While you can argue this should not be used as production setup, > > this can be totally valid for sandboxes and we wanna provide > > consistent and reliable behavior for users. > > > > But my argument was not in if/how we want to fine-tune > > deployments, but also understand and provide means to define > > what's needed as well as potential ability to revert in > worst case > > scenario as a temporary workaround. > > So still some variables and logic would be introduced from > what I > > understand today. > > > > > > On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> > wrote: > > > > > > On 16/06/2025 13:27, Dmitriy Rabotyagov wrote: > > > > > > > > > > > > sayint its FUD is not helpful. > > > > > > we got a driect ask form operator and soem core to not > > do a hard > > > switch > > > over. > > > > > > and while i wanted to only support one model for each > > binary at a > > > time > > > we were sepcificly ask to make it configurable. > > > > > > > In the later case, your only available action is > help > > fixing > > > bugs. It > > > > is not up to the operators to double-guess what > may or > > may not > > > work. > > > > > > correct we are not planning to document how to change > > mode we were > > > planning to only use this configuration in ci and > > operator would be > > > > > > > > > Well, we'd need to have that communicated so that > deployment > > toolings > > > could adopt their setup to changes, as, for instance, > in OSA > > amount of > > > eventlet workers are calculated based on the system facts, > > so we'd > > > need to change the logic and also suggest how users should > > treat this > > > new logic for their systems. > > > > why is OSA doing that at all today? > > we generally don't recommend changing those values from the > > default > > unless you really know what your doing. > > i don't think other installer do that. > > tripleo, kolla-ansbile and our new golang based installer do > > not, nor > > does devstack so its surprising to me that OSA would change > > such low > > level values > > by default. > > > > we will document any new config options we and and we are > > documentation > > how to tune the new options for thread pools but we do > not expect > > installation > > tools to modify them by default. we are explicitly not > making the > > options based on the amount of resources on the host i.e. > > dynamically > > calculated based > > on the number of CPU cores. > > > > for example we are explicitly setting the number of > > scatter_gather > > thread in the the dedicated thread pool to 5 > > why its a nice small number that will work for most > people out > > of the box. > > > > can you adjust it, yes but it scale based on the number of > > nova cells > > you have an 99% wont have more then 5 cells. > > > > using information about the host where the API is > deployed to > > infer the > > value of that would be incorrect. > > > > you can really only make an informed decision about how to > > tune that > > based on monitoring the usage of the pool. > > > > that how we expect most of the other tuning options to > go as well. > > > > our defaults in nova tend to be higher then you would > actually > > need in a > > real environment so while it may make sense to reduce > > them we try to make sure the work out of the box for > most people. > > > > gibi id building up > > > https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co… > > > > as part of nova move to encode this but our goal is that > > deployment > > tools shoudl not need to be modifyed to tune these > > valued by defualt. > > > > > > > > So it will be kinda documented in a way after all. > > > > > > > > > told for a given release deploy this way. > > > > > > this is an internal impelation detail however we > are not > > prepared to > > > deprecate usign eventlet until we are convicned > > > > > > that we can run properly without it. > > > > > > > For beginners, this would be a horrible nightmare if > > default > > > options > > > > simply wouldn't work. We *must* ship OpenStack > working > > by default. > > > no one is suggesting we do otherwise. > > > > > > > > Cheers, > > > > > > > > Thomas Goirand (zigo) > > > > > > > > > > > > > -- > Hervé Beraud > Principal Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ >

1 month, 4 weeks

Re: [oslo] Remove eventlet from openstack.

by Herve Beraud

> i was expeict that oslo.service would not be released as part of the > eventlet removal > and instead would have been ported to not depend on eventlet. > > i.e. have oslo.service perodici tasks just delegate to futurist > providing the same api with a diffent backend. > > so i was kidn of surprised to see this topic come up. > I'll focus my answer only on that point ^. We were not thinking of managing the oslo.service topic at the same time as the Eventlet removal, but unfortunately, both topics can be separated... As oslo.service is heavily based on Eventlet, we can't touch one without touching the other. We were thinking of giving orientation for replacement through deprecation messages and through guidance [1]. But, indeed, a binding to futurist and cotyledon through oslo.service could be an option. Indeed it would really simplify the migration, but on the other hand it will produce technical debt, which at term, risks to remain an undefined amount of time. If people feel comfortable using this binding, they won't have reasons to drop oslo.service in favor of futurist and cotyledon. They will continue using the binding... But this side effect doesn't seem so terrible, it could be a credible scenario in the long term, so I don't think we should ignore that point. But if we consider the benefits brought by such binding, this side effect could be surely ignored. Indeed, on the other hand, such a kind of binding would be a good option to: 1. not have to internalize cotyledon into OpenStack. I'm personally more in favor of keeping Cotyledon as an independent deliverable as we did with Eventlet; 2. to host a wsgi binding (maybe based on flask) as an alternative to the Eventlet WSGI binding offered by oslo.service. Maintaining the binding would allow us to monitor if Cotyledon needs help at some point. Maintaining a binding would simplify the migration. Maintaining a binding could merge the best of both worlds in many aspects. [1] https://709b9532d61399a9e320-55e7a9d33a731efe4f7611907a31a4a1.ssl.cf2.rackc… -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 year, 2 months

Re: Upstream Oslo IRC Meetings Are Back.

by Takashi Kajinami

On 11/4/24 6:00 PM, Herve Beraud wrote: > > > Le jeu. 31 oct. 2024 à 18:38, Takashi Kajinami <kajinamit(a)oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>> a écrit : > > I noticed that eventlet removal meeting is independent from > the oslo meeting proposed in this thread. I'm ok to have dedicated > slot to discuss oslo specific topics but may need to have clear scope > definition. (Since it was mentioned that eventlet removal is the core > topic we want to discuss in this meeting, right ?). > > > Oslo meetings are Oslo meetings, isn't the lifecycle of Oslo and the Oslo deliverables a clear scope? Oslo owns more than 40 deliverables, I don't think it would be difficult to find Oslo things to discuss. Another option would be to follow the series agenda, and to base our Oslo meetings over the series lifecycle (milestones, FF, etc.). > > Are people supposed to use this meeting agenda (https://wiki.openstack.org/wiki/Meetings/Oslo <https://wiki.openstack.org/wiki/Meetings/Oslo>) to put their topics and to retrieve the topics you propose? > > If you propose to base our discussions on a series lifecycle, maybe it could worth basing our meetings on an agenda like the one used by the release team https://etherpad.opendev.org/p/epoxy-relmgt-tracking,The <https://etherpad.opendev.org/p/epoxy-relmgt-tracking,The> release team created tooling to generate this kind of etherpad, perhaps you could reuse it... I totally agree with you about these points and these are exactly what I'd propose for oslo specific meeting. I just wanted to make sure that Daniel's expectation is similar, because in his initial email sounded that the eventlet removal is the key motivation to restore the meeting. > Concerning Eventlet. Eventlet may impact Oslo in some ways, but Oslo meetings are not the right place to host all the Eventlet discussions. Eventlet discussions will cover a wide range of topics, from neutron, to packaging, passing by requirements, and surely also Oslo at some points, so I don't think Oslo meetings is the right place. > Nova, Neutron, Glance and surely many other deliverables already have internal discussions about Eventlet in their own IRC meetings, IMO Oslo should follow the same path. +1 > Regarding the time slot, EMEA/NA slots might work better for me because > I can join these after I settle down my own things, but I think it's > generally got to have more Asia friendly slots if anyone from Asia > (especially east-Asia) is interested in joining us. > > > You are the chair [1] of the Oslo meeting, so "joining us" mostly means "joining you". Asia time slots are before EMEA/NA so if you have your "own things before", my question is: would you be able to join and manage the Asia time slot? If your answer is no, IMO, it is useless to propose an Asia time slot, as people will wait for the meeting chair. This is the question to Daniel who is the primary chair (though I can help when he is not available on best effort basis). I wanted to make you aware of my preference assuming you may count me as a participant from Asia. > > It is up to you. > > [1] https://meetings.opendev.org/#Oslo_Team_Meeting <https://meetings.opendev.org/#Oslo_Team_Meeting> > > > On 10/31/24 7:07 PM, dbengt(a)redhat.com <mailto:dbengt@redhat.com> wrote: > > > > > > On 10/30/24 5:57 PM, Ghanshyam Mann <gmann(a)ghanshyammann.com <mailto:gmann@ghanshyammann.com>> wrote: > >> ++ on alternate TZ meetings, and that way, EU and Asia TZ can be covered in one of the meetings. > >> To keep it simple, we can have separate doodle poll for each TZ meeting. > > Ok so maybe instead we can have one meeting per week. As you proposed like one for the Europeean and Asia and one for the us? I have create a first UTC pool here[1] afternoon and here[2] you have the morning one. > > > > [1] https://www.vyte.in/e/1cnp51e4sd <https://www.vyte.in/e/1cnp51e4sd> [2] https://www.vyte.in/e/2j0to257cd <https://www.vyte.in/e/2j0to257cd> > > > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ <https://github.com/4383/> >

9 months, 1 week

[neutron] Bug deputy report for week of February 24th

by Brian Haley

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * None High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2100001 - [FT] Random errors in ``test_server.TestPluginWorker.test_start`` - https://review.opendev.org/c/openstack/neutron/+/942798 * https://bugs.launchpad.net/neutron/+bug/2100261 - [FT] Consistent errors in ``neutron-functional-with-pyroute2-master`` CI job - https://github.com/svinota/pyroute2/issues/1181 might be related - Rodolfo took ownership * https://bugs.launchpad.net/neutron/+bug/2100270 - [whitebox-neutron-tempest-plugin] Missing method ``configure_vlan_transparent`` imported from neutron-tempest-plugin - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/942776 * https://bugs.launchpad.net/neutron/+bug/2099981 - [ovn] Adding an IPv6 subnet to public network breaks all FIPs - Still trying to reproduce * https://bugs.launchpad.net/neutron/+bug/2099982 - [ovn] Removal of IPv6 subnet from the public network is not reflected in LRs - Might be related to above OVN bug 2099981 * https://bugs.launchpad.net/neutron/+bug/2100294 - [ovn] Changing gateway for IPv6 subnet updates wrong routes family - Might be related to above OVN bug 2099982 Medium ------ * https://bugs.launchpad.net/neutron/+bug/2099911 - Castellan as config source fails with "greenlet.error: cannot switch to a different thread" - https://review.opendev.org/c/openstack/neutron/+/942622 - But don't think that might be necessary with us removing eventlet * https://bugs.launchpad.net/neutron/+bug/2100505 - Neutron OVN external ports creation change HA chassis group master - FT reproducer patch confirms issue - Needs owner Low --- * https://bugs.launchpad.net/neutron/+bug/2100479 - Segments plugin: Not updating used IP addresses - https://review.opendev.org/c/openstack/neutron/+/935990 might fix? Wishlist -------- * https://bugs.launchpad.net/neutron/+bug/2099896 - [eventlet-removal] Remove the usage of eventlet in the Metadata agent - Needs owner * https://bugs.launchpad.net/neutron/+bug/2100268 - cleanup os-ken from the CLI part - Needs owner * https://bugs.launchpad.net/neutron/+bug/2100585 - [metadata] Unify metadata code - Eventlet follow-up

5 months, 1 week

Re: [eventlet-removal][release][TC] When to drop eventlet support

by Herve Beraud

I think there is an option C. Since the beginning of this initiative, the goal has always been to phase out Eventlet cleanly from OpenStack components, particularly Nova. The initial proposal targeted the 2027.2 release, at a time when Nova had no plans to support a dual concurrency model. At that point, Eventlet removal was not classified as a feature removal subject to SLURP policy - it was considered an internal refactoring, with no addition of transitional features requiring later removal. The idea was to deprecate any Eventlet-related features, public APIs, and configuration options as early as possible. As a result, SLURP policy constraints were not a primary concern. But the situation has evolved. Nova has now committed to offering a full release cycle (2027.1) where both concurrency models - Eventlet and threading - are supported. This is not a technical convenience, but a response to explicit operator requests: operators want the ability to switch concurrency modes outside in case of problem during at least one series. The full threading support in Nova will not be ready before 2026.2, which is a non-SLURP release, and that the dual-mode support must be preserved throughout 2027.1. However, current SLURP policy requires that a removal can only happen in a SLURP release. According to Nova's plan that excludes a removal in 2027.1. If we follow the rules strictly, Eventlet removal would be delayed to 2028.1, adding several months of dual-mode support, increased maintenance complexity, and growing technical debt. From a technical standpoint, it’s important to highlight that upcoming Python versions (3.14 and 3.15) introduce major changes to the GIL [1], threading behavior, and RLocks - all areas [2] that have historically been problematic in Eventlet. We’ve already had to implement significant patches for Python 3.13. There is also uncertainty around which Python versions will be adopted by the various Linux distributions and by our own supported runtime environments, making long-term compatibility assumptions with Eventlet increasingly risky, but in all the cases it will put additional pressure on us. Continuing to support Eventlet until 2028 is effectively betting on the stability of a module that is already on life support, with very few contributors available to maintain it beyond 2027 (see Jay's note about GR-OSS). More critically, there is no guarantee today that Eventlet, in its current state, will remain robust enough to reliably support a fallback path in case a regression forces a rollback from threading to Eventlet. This is why I propose a pragmatic and balanced compromise: we maintain the planned deprecation (we might have to adjust a little bit those from oslo), but we make a documented and limited exception to allow Eventlet removal in 2027.2, which is a non-SLURP cycle. This exception would be coordinated with the Technical Committee and the release team and is justified by: - the commitment made by Nova to offer a full cycle of dual-mode support in 2027.1; - the high technical cost of maintaining dual concurrency models; - the fast evolution of Python and the increasing risk of incompatibilities with Eventlet; - the limited engineering resources available to sustain Eventlet maintenance; - and the need to align with operator expectations while still securing the long-term health of the platform. This approach gives Nova the entire 2027.1 cycle to deliver a complete and stable dual-mode implementation, as promised to operators. At the same time, it enables the rest of the ecosystem to move forward safely, without relying on a dependency that is increasingly brittle and obsolete. In large open source projects, it is not uncommon for policies to be temporarily adjusted in the face of structural or technical emergencies. This is not a convenience exception - it’s a reasoned and responsible deviation from the norm, driven by necessity. For this reason I add the release team and the TC to the party. Please share your feedback with us. [1] https://docs.python.org/es/dev/whatsnew/3.14.html#whatsnew314-free-threaded… [2] https://docs.python.org/es/dev/whatsnew/3.15.html#threading. Le lun. 16 juin 2025 à 18:06, Sean Mooney <smooney(a)redhat.com> a écrit : > > On 16/06/2025 15:33, Dmitriy Rabotyagov wrote: > > Also let's keep in mind, that only nova (with placement) will spawn 64 > > threads on such setup by default. > > yep see my other reply. > > you mixign up workers and eventlet threads. > > those are two very diffent things. > > we likely shoudl change the upstream default for workers to 1. > > we do tend to override that by defualt in installer tools. > > > > > And then all really depends on set of services to launch on such setup. > > > > So from deployment tooling protective you have all required data to > > rollout not instantly oom-ing setup at cost of amount of request > > services can process in parallel. > > > > On Mon, 16 Jun 2025, 15:24 Dmitriy Rabotyagov, > > <noonedeadpunk(a)gmail.com> wrote: > > > > In case you try to use a 32gb box with 16 cores as a controller > > for OpenStack - it will blow off with default amount of workers > > for wsgi and /or eventlet apps. > > > > While you can argue this should not be used as production setup, > > this can be totally valid for sandboxes and we wanna provide > > consistent and reliable behavior for users. > > > > But my argument was not in if/how we want to fine-tune > > deployments, but also understand and provide means to define > > what's needed as well as potential ability to revert in worst case > > scenario as a temporary workaround. > > So still some variables and logic would be introduced from what I > > understand today. > > > > > > On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote: > > > > > > On 16/06/2025 13:27, Dmitriy Rabotyagov wrote: > > > > > > > > > > > > sayint its FUD is not helpful. > > > > > > we got a driect ask form operator and soem core to not > > do a hard > > > switch > > > over. > > > > > > and while i wanted to only support one model for each > > binary at a > > > time > > > we were sepcificly ask to make it configurable. > > > > > > > In the later case, your only available action is help > > fixing > > > bugs. It > > > > is not up to the operators to double-guess what may or > > may not > > > work. > > > > > > correct we are not planning to document how to change > > mode we were > > > planning to only use this configuration in ci and > > operator would be > > > > > > > > > Well, we'd need to have that communicated so that deployment > > toolings > > > could adopt their setup to changes, as, for instance, in OSA > > amount of > > > eventlet workers are calculated based on the system facts, > > so we'd > > > need to change the logic and also suggest how users should > > treat this > > > new logic for their systems. > > > > why is OSA doing that at all today? > > we generally don't recommend changing those values from the > > default > > unless you really know what your doing. > > i don't think other installer do that. > > tripleo, kolla-ansbile and our new golang based installer do > > not, nor > > does devstack so its surprising to me that OSA would change > > such low > > level values > > by default. > > > > we will document any new config options we and and we are > > documentation > > how to tune the new options for thread pools but we do not expect > > installation > > tools to modify them by default. we are explicitly not making the > > options based on the amount of resources on the host i.e. > > dynamically > > calculated based > > on the number of CPU cores. > > > > for example we are explicitly setting the number of > > scatter_gather > > thread in the the dedicated thread pool to 5 > > why its a nice small number that will work for most people out > > of the box. > > > > can you adjust it, yes but it scale based on the number of > > nova cells > > you have an 99% wont have more then 5 cells. > > > > using information about the host where the API is deployed to > > infer the > > value of that would be incorrect. > > > > you can really only make an informed decision about how to > > tune that > > based on monitoring the usage of the pool. > > > > that how we expect most of the other tuning options to go as > well. > > > > our defaults in nova tend to be higher then you would actually > > need in a > > real environment so while it may make sense to reduce > > them we try to make sure the work out of the box for most people. > > > > gibi id building up > > > https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co… > > > > as part of nova move to encode this but our goal is that > > deployment > > tools shoudl not need to be modifyed to tune these > > valued by defualt. > > > > > > > > So it will be kinda documented in a way after all. > > > > > > > > > told for a given release deploy this way. > > > > > > this is an internal impelation detail however we are not > > prepared to > > > deprecate usign eventlet until we are convicned > > > > > > that we can run properly without it. > > > > > > > For beginners, this would be a horrible nightmare if > > default > > > options > > > > simply wouldn't work. We *must* ship OpenStack working > > by default. > > > no one is suggesting we do otherwise. > > > > > > > > Cheers, > > > > > > > > Thomas Goirand (zigo) > > > > > > > > > > > -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 month, 4 weeks

Re: [eventlet-removal]When to drop eventlet support

by Dmitriy Rabotyagov

On Sat, 14 Jun 2025, 01:24 , <thomas(a)goirand.fr> wrote: > > > Right. Plus I don't get why operators get to choose what class of bugs > they may experience, and how they will know beter than contributors. > Cheers, > Well, at least some class of bugs might be already known and operators might be willing to accept them, while the new ones could be not known by project maintainers and take quite some time to realize and patch. So being able to rollback to old behavior might save the day. And an extremely good example for that is heartbeat_in_pthread, from oslo.messaging, for which we've flipped default one day and mark it as deprecated, untill realized it had a terrible consequences for WSGI applications a couple of cycles later, so default had to be flipped back, while operators were dealing/work-arounding with consequences by changing the option back and selecting old buggy over the new ones, as the service was still working more reliably with old ones. Thomas Goirand (zigo) > >

2 months

openstack-discuss search results for query "#eventlet-removal"