openstack-discuss search results for query "#eventlet-removal"
openstack-discuss@lists.openstack.org- 186 messages
[glance] Flamingo PTG summary
by Cyril Roelandt
Hi,
We have just had our virtual PTG. The full list of topics, along with
notes, can be found on our Etherpad[1]. Here is a summary of the topics
that will be targeted during this cycle.
If you would like to view one of the recordings, please reach out to us
so we can figure out a way to share it.
# glance-tempest-plugin work
There has not been a lot of activity on the Glance Tempest Plugin for a
while. We decided to put more effort into it this cycle. The first step
is to decide what stale patches to abandon or rebase. We will also try
to add tests for newer APIs added in the past cycles (location, quota,
cache).
# Eventlet removal
While Glance can already be deployed in production without eventlet,
this library is still required for half of our functional tests, as well
as for non-core features (namely the scrubber). We will be focusing on
migrating our functional tests away from eventlet during this cycle.
# Image encryption
Not much progress has been made on this topic in the past cycle. The
original authors of this feature are willing to spend time working on
the required patches during the Flamingo cycle; the Glance core dev will
make it a priority to quickly review the required patches so that we can
finally merge the feature.
# Glanceclient chunked transfer
By default, the Glance client transfers data to the Glance API in
chunks. This may force the Cinder backend to resize the volume multiple
times, slowing down the upload. Providing the total size of the image
will allow us to fix this issue.
# Cinder common configuration options
Operators often have parameters set to the exact same values for
multiple Cinder backends. We plan on making it possible to configure
"common configuration options", thus allowing operators not to repeat
configuration snippets.
# Freeze glanceclient development
As we approach feature parity between the Glance client and the
OpenStack unified client, we believe it will be best to freeze
development of the Glance client. In the future, we will only fix bugs
and security issues. During this cycle, we will work on actually
reaching feature parity with the unified client.
# new location API (Cinder & Nova)
Glance has introduced two new Location APIs in the Dalmatian cycle. We
can use these APIs to address OSSN-0090 and OSSN-0065. Patches for Nova
and Cinder must still be merged, hopefully during the Flamingo cycle.
# freeze glance client development
Cinder and Nova will need to use the OpenStack SDK instead of the
glanceclient. There is no need to complete this work during this cycle,
but we should at least have a good idea of what APIs are currently being
used, so that we can have a plan for the next cycles.
# cache-cleaner/pruner
We discussed possible minor improvements to both the cache-pruner and
the cache-cleaner. This work is not going to be a priority for this
cycle.
# Deprecate the filesystem_store_datadirs configuration option
This option has not been needed since the introduction of multistore
support.
Happy Hacking!
Cyril Roelandt
[1] https://etherpad.opendev.org/p/apr2025-ptg-glance
8 months, 2 weeks
Re: Eventlet and debugging
by smooney@redhat.com
On Wed, 2024-08-14 at 23:06 +0100, smooney(a)redhat.com wrote:
> On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote:
> > Thanks to both of you for the response . I try to insert pdb breakpoint at
> > some point in the code files and start the service. Then I issue a nova
> > command to check how the flow works. From then on I use the pdb commands to
> > check the function calls. This to me gives a very convenient way of
> > learning the code connections, since I don't have an idea of how these
> > functions and modules r related. U people as developers have a design to
> > start with and work on from there. So u may know the purpose of each
> > function, class, etc.
> >
> > Since nova api doesn't support pdb becos of eventlet design, I learned to
> > print those objects and find their information.
> >
> > But with pdb, it's a smooth flow to inspect the class attributes, their
> > functions, arguments, types etc. I can even test those objects the python
> > way since pdb supports python prompt as well. Of course I do all this on
> > dev systems though...
>
> this is going to be a long responce and its a bit rambely but im goign to provide
> you with the context and poitn er try and figure out how to do this for your self.
>
> first thing to note is while its possibel it not trivial and the payoff is proably not
> worth it in the long run. we do not have up to day docs for how to do this because
> this is not how peole learn, contibute to ro develop nova in genral.
>
>
> nova does have a remote debug facilitate that is sepreate for the eventlet backdor to enable
> you to use an ide to debug remotely.
> that was added around 2012 and has worked on an off to vering degrees since then.
>
> the first thing to understand about nova and several other project is that they are a distibuted system
> i.e. the applciaiton that compise a given service like nova run in multiple service which are interconnected
> vai a message bus with in a service and rest apis are exposed for inter service comunication.
>
> when you enable a debugger and stop on a breakpoitn only one of the process is stop and the other contiue
> to exsculte which means that RPC calls, heathbeats and other asynconos tasks can and will timeout and fail.
>
> you cannot debug a distibuted system in teh same way as you woudl a simple applction where all state
> is executed in a signle main loop.
>
> the act of observing the internal behavior of the nova api or nova-comptue agent will change its behavior.
>
> nova has suppoort for remote debugging that was added about 12 years ago
> https://wiki.openstack.org/wiki/Nova/RemoteDebugging
> https://github.com/openstack/nova/blob/master/nova/debugger.py
>
> with that supprot you can use an idea like pycharm to connect to the remote debugger and attempt to
> debug a process.
>
> im currently propsoing removing that as part of the removal of eventlet partly because it shoudl not be needed
> when we dont have enventlet and mainly because it didnt work very well when i last used it.
>
> some ides have supprot fo gevent
>
> gevent is a fork of eventlet which was forked form some of hte libiares in stackless python
> eventlet converts a single thread python interperater in a cooperative concurent multi userland process.
>
by the way i forgot to say if you want to debug eventlet mokey patch code
adn your using a debugger that supprot gevent it mostly work with eventlet too if you enabel the gevnet
supprot in pycham which is disabeld by defaut.
i have had it workk reallly well for months and then not work for years and suddenly start workign agian.
so the proablme with usign a debugger with nova is not getting it working initally its keepign it working
and setting it up on a new machine if you create a new devstack vm for dev again.
its not hard, its hard to do repatbly, so blogs, docs and even automation
tends to bit wroth quickly.
> what does that mean , each python function is not operating in a seperate os stackframe, it has heap allcoated
> stack frames that allow context swtichs between userland thread at any tiem a funnction woudl do blocking io
>
> the reason single step debugging does not work well with gdb is that when you single step the eventlet engine
> is often going to end up context switching into the midel of another function.
>
> to work around this you just set breakpoint at the line you want to go to next and use continue instead fo single step
>
> one of the rules when using eventlet is that if your going to monkey patch you need to monkey patch everyting early
>
> in addtion too the remote debuging facilites we can disabl mokey patching entirly.
> you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87
>
> this will entirly break nova-compute and several other nova process as they have
> effectivly infinity loops to pool for rpc messags, monitor hyperviors ectra.
>
> if the remote debugger is used it disables patching the thread lib
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46
>
> that is prartly why usign the remote debugger is kind fo buggy in itsself.
> the code is not inteded to work that way.
>
> these fucntionlatiy mainly exist to allow debuging tests not the actulaly running code.
>
> if i need to debug test i normaly jsut comment out
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89
> and run the test in my ides(pycharm,vscodes) test debbuger.
> i normally code in nano/emacs but is do use ides if i really cant figure out what happening
> however i genreally start with just adding log lines.
>
> when i first started workign on opentack in 2013 i very much liked using an ide and debugger
> and i woudl often spend time trying to get nova to work in a debugger.
>
> over tiem i generally found that that was not worth the effort.
> keepign a devstack deployment working that way was often more work then doing print staement
> or better writing functional or unit test to repoduce the problem.
>
> you previous mails ask about debuging nova's api.
>
> if you want to do that its one of the easiest thigns to debug.
> first do _not_ use the wsgi script
> adding apache of modwsgi is just going to make your life harder.
>
> just disable eventlet monkey patching, by commetiing or usign the env varible
> and run the nova-api console script directly under pdb or your debugger of choice.
> that will allow you to basiclly single step debug without any issues as it will not be using eventlet
> anymore and it has not infinitly loops or perodic tasks so it mostly just works when not using eventlet.
>
> that becase it didnt use eventlet at all in the api until about 2016 and even today it only really uses it in one
> place
> in my eventlet removal serise i split nova in to the part that use eventlet directly and the parts that dont in the
> second path
>
> https://review.opendev.org/c/openstack/nova/+/904424
>
> anything under nova/cmd/eventlet/ needs eventlet today
> everything under nova/cmd/standalone/ does not.
>
> the approch of doign an api call and following it in a debugger is not a good way in my expericne to learn how
> somethign
> like nova really works. it can hellp to a limtied degree but you cant eaislly debug across multipel processes.
> a minium nova deployment has 4 process typically more.
>
> you can try but each openstack service (nova, neutorn, cinder, ...) is typically compised of several micocservices
> some like keystone and palcement are just rest apis in front of a db and dont use eventlet at all so are trivial to
> debug as a result. the rest have non tirival runtime interactions that cant be as eaislly decompsed.
> becuase of that nova and other project invest in our functional tests to allwo ues to emulate that multi process env
> and better reason about the code.
>
> the debugging functionality is documetned in the contibutor guide
> https://github.com/openstack/nova/blob/master/doc/source/contributor/develo…
> but we dont really adverstise that because none of the core maintainers use that any more and it has not been
> maintained
> for the better part of a decade. since we cant use this type of debugging with our ci system
> if we cant debug why a failure happens form the ci logs we add more logs because thyat will be useful for future us
> and operators in production. so our focus is on logging/monitoring for production rather then for developer eases
> simply because of the large techdebt that eventlet creates in this aready.
>
> i know nuetorn has had some better look with using debuggers then nova has,
>
> keystone/placmnet are eventlest free adn are just python web apps/rest apis so should "just work"
> in any python debugger.
>
> nova is prehaps the hardest to debug because of the legacy of the proejct so its not the best palce to start looking
> how this works. too be clear i have ran nova api, nova-conductor, nova scheduler and nova-compute all in 4 ide windows
> with 4 debuggers at once
> it was proably 3 year of working on openstack before i understood how to hack that ot work and since i learned
> how to properly debug without a debuggger, logs, code inspection, unit/fucntioal testing
> i have nver really need to do that again.
>
>
> gaining the ablity to run openstack in a debugger simply again, in a maintainable way is deffienly one of the reason
> im
> looking forward to removing eventlet but its still a lot of work.
> im not sure how helpful this was but this is proably as clsoe to a blog post as you will get.
>
> >
> > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote:
> >
> > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote:
> > > > How many more releases are you planning to include eventlet in
> > > > them? Seems they are removing any support for further development
> > > > according to their official website. So, is the community thinking
> > > > of any alternatives?
> > > [...]
> > >
> > > Just to expand on the first part of Sean's reply, most of what you
> > > want to know is probably answered here:
> > >
> > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html
> > >
> > > --
> > > Jeremy Stanley
> > >
>
1 year, 4 months
[cinder][PTG] 2026.1 Gazpacho PTG summary
by Jon Bernard
Hello everyone, thanks for a very productive PTG for the Gazpacho cycle.
Below is a brief summary of the highlights. The raw etherpad notes are
here [1] and both the summary and transcribed etherpad notes are
captured in out wiki here [2].
# Retrospective
We spent time discussing the good and not-so-good of the flamingo cycle.
Unsurprisingly, code reviews continue to be our most significant
opportunity for improvement. We reviewed what a core reviewer is and
how to work towards becoming one, including the metrics that we use to
determine eligibility and what we look for in a potential core reviewer.
I encourage anyone interested in contributing the project to reach out,
in addition to our guideline document on the review process, we're happy
to answer questions and help guide folks through the process.
# Festival of Reviews
We are reworking our 'festival of reviews' meeting to better accommodate
a larger set of potential patches. Previously it focused on very small
patches that were easier to review, but feedback suggests that time
spent as a team on larger reviews would yield a more desirable result.
Last Friday we ran the modified format and it seemed to work quite well.
# 3rd Party CI
This cycle 3rd party CI is required for patches to be reviewed. In the
flamingo cycle we saw some challenges with deploying and configuring the
software factory setup required to operate a CI. We remain committed to
helping vendors sort out any remaining problems, and want to avoid any
further slipping on our requirements, as one-off exceptions do not
scale.
# Eventlet Removal
There will be significant focus this cycle on the removal of eventlet
from our codebase. Our goal is to have this work completed before M2.
The remaining issues and technical challenges were discussed, including
some struggles with our multi-backend and it's use of sqlalchemy. We
learned that the oslo team is working on some code that may help with
this effort. We may have finally found a use for the rally job,
unfortunately it has been broken for some time, we will try to get this
fixed and reporting useful results this cycle as well.
# Driver Block Geometry
An original patch to the RBD driver was proposed to make configuring
block geometry an option for the operator. Upon inspection, we decided
make this option available for all drivers. Discussion about exact
location given out driver class model, we have a way forward.
https://review.opendev.org/c/openstack/cinder/+/658283
# Per-Tenant SVM Creation
Doug Goldstein showed up and blew our minds with a proof-of-concept
feature he added to the NetApp driver. This change allows the driver to
create per-tentant SVMs and map them to dynamically created backends.
There will be some technical hurdles to overcome, but the idea in theory
could work. Doug is going to work to propose a patch upstream and with
the NetApp team to see if it can take shape.
# OpenAPI
Stephen has had patches up for openAPI compliance for far too long. We
don't have any technical objections to the effort. Brian and I
committed to working through the proposed patches at the very start of
the cycle. Significant progress already, should be completed soon.
# Reverting To Any Snapshot
Currently you can only roll a volume back to the last available
snapshot. We would like to be able to select any snapshot as a rollback
target. Technical requirements were discussed, including the need for a
spec. Hoping to see progress on this effort in this cycle.
# Clone Across Pools
We are interested in improving our implementation for cloning a volume
across different pools. The code itself has some distinct shortcomings
and Rajat identified several opportunities for improvement. Feedback on
current usage was requested, we discussed volume caching as part of
these improvements. A path forward was defined and spec should follow.
# NFS Driver Feedback
Fernando has been working on several improvements to the generic NFS
driver and requested feedback. NetApp offered valuable suggestions,
some improvements to our tempest testing patches are forthcoming, and a
direction forward is understood.
# Replication Feedback
We solicited feedback on replication, its current usage, and the
features it may require to see wider adoption. Questions and answers
were discussed, hoping to see a spec capturing the most valuable changes
in this cycle.
[1] https://etherpad.opendev.org/p/cinder-gazpacho-ptg
[2] https://wiki.openstack.org/wiki/CinderGazpachoPTGSummary
--
Jon
1 month, 3 weeks
Re: Eventlet and debugging
by Mike Carden
In almost a decade of reading smooney(a)redhat.com's Lysdexic posts, this is
the longest one I have ever read. No criticism intended at all - none
whatsoever.
BUT... I am still absolutely Gobsmacked that anyone can possibly write even
vaguely working code, if their written English communications are so
comprehensively broken. How does anyone get away with ignoring Spelling and
Grammar in Code?
I love your work Sean, but I do struggle to understand almost everything
you type.
--
MC
On Thu, 15 Aug 2024 at 08:07, <smooney(a)redhat.com> wrote:
> On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote:
> > Thanks to both of you for the response . I try to insert pdb breakpoint
> at
> > some point in the code files and start the service. Then I issue a nova
> > command to check how the flow works. From then on I use the pdb commands
> to
> > check the function calls. This to me gives a very convenient way of
> > learning the code connections, since I don't have an idea of how these
> > functions and modules r related. U people as developers have a design to
> > start with and work on from there. So u may know the purpose of each
> > function, class, etc.
> >
> > Since nova api doesn't support pdb becos of eventlet design, I learned to
> > print those objects and find their information.
> >
> > But with pdb, it's a smooth flow to inspect the class attributes, their
> > functions, arguments, types etc. I can even test those objects the python
> > way since pdb supports python prompt as well. Of course I do all this on
> > dev systems though...
>
> this is going to be a long responce and its a bit rambely but im goign to
> provide
> you with the context and poitn er try and figure out how to do this for
> your self.
>
> first thing to note is while its possibel it not trivial and the payoff is
> proably not
> worth it in the long run. we do not have up to day docs for how to do this
> because
> this is not how peole learn, contibute to ro develop nova in genral.
>
>
> nova does have a remote debug facilitate that is sepreate for the eventlet
> backdor to enable
> you to use an ide to debug remotely.
> that was added around 2012 and has worked on an off to vering degrees
> since then.
>
> the first thing to understand about nova and several other project is that
> they are a distibuted system
> i.e. the applciaiton that compise a given service like nova run in
> multiple service which are interconnected
> vai a message bus with in a service and rest apis are exposed for inter
> service comunication.
>
> when you enable a debugger and stop on a breakpoitn only one of the
> process is stop and the other contiue
> to exsculte which means that RPC calls, heathbeats and other asynconos
> tasks can and will timeout and fail.
>
> you cannot debug a distibuted system in teh same way as you woudl a simple
> applction where all state
> is executed in a signle main loop.
>
> the act of observing the internal behavior of the nova api or nova-comptue
> agent will change its behavior.
>
> nova has suppoort for remote debugging that was added about 12 years ago
> https://wiki.openstack.org/wiki/Nova/RemoteDebugging
> https://github.com/openstack/nova/blob/master/nova/debugger.py
>
> with that supprot you can use an idea like pycharm to connect to the
> remote debugger and attempt to
> debug a process.
>
> im currently propsoing removing that as part of the removal of eventlet
> partly because it shoudl not be needed
> when we dont have enventlet and mainly because it didnt work very well
> when i last used it.
>
> some ides have supprot fo gevent
>
> gevent is a fork of eventlet which was forked form some of hte libiares in
> stackless python
> eventlet converts a single thread python interperater in a cooperative
> concurent multi userland process.
>
> what does that mean , each python function is not operating in a seperate
> os stackframe, it has heap allcoated
> stack frames that allow context swtichs between userland thread at any
> tiem a funnction woudl do blocking io
>
> the reason single step debugging does not work well with gdb is that when
> you single step the eventlet engine
> is often going to end up context switching into the midel of another
> function.
>
> to work around this you just set breakpoint at the line you want to go to
> next and use continue instead fo single step
>
> one of the rules when using eventlet is that if your going to monkey
> patch you need to monkey patch everyting early
>
> in addtion too the remote debuging facilites we can disabl mokey patching
> entirly.
> you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87
>
> this will entirly break nova-compute and several other nova process as
> they have
> effectivly infinity loops to pool for rpc messags, monitor hyperviors
> ectra.
>
> if the remote debugger is used it disables patching the thread lib
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46
>
> that is prartly why usign the remote debugger is kind fo buggy in itsself.
> the code is not inteded to work that way.
>
> these fucntionlatiy mainly exist to allow debuging tests not the actulaly
> running code.
>
> if i need to debug test i normaly jsut comment out
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89
> and run the test in my ides(pycharm,vscodes) test debbuger.
> i normally code in nano/emacs but is do use ides if i really cant figure
> out what happening
> however i genreally start with just adding log lines.
>
> when i first started workign on opentack in 2013 i very much liked using
> an ide and debugger
> and i woudl often spend time trying to get nova to work in a debugger.
>
> over tiem i generally found that that was not worth the effort.
> keepign a devstack deployment working that way was often more work then
> doing print staement
> or better writing functional or unit test to repoduce the problem.
>
> you previous mails ask about debuging nova's api.
>
> if you want to do that its one of the easiest thigns to debug.
> first do _not_ use the wsgi script
> adding apache of modwsgi is just going to make your life harder.
>
> just disable eventlet monkey patching, by commetiing or usign the env
> varible
> and run the nova-api console script directly under pdb or your debugger of
> choice.
> that will allow you to basiclly single step debug without any issues as it
> will not be using eventlet
> anymore and it has not infinitly loops or perodic tasks so it mostly just
> works when not using eventlet.
>
> that becase it didnt use eventlet at all in the api until about 2016 and
> even today it only really uses it in one place
> in my eventlet removal serise i split nova in to the part that use
> eventlet directly and the parts that dont in the
> second path
>
> https://review.opendev.org/c/openstack/nova/+/904424
>
> anything under nova/cmd/eventlet/ needs eventlet today
> everything under nova/cmd/standalone/ does not.
>
> the approch of doign an api call and following it in a debugger is not a
> good way in my expericne to learn how somethign
> like nova really works. it can hellp to a limtied degree but you cant
> eaislly debug across multipel processes.
> a minium nova deployment has 4 process typically more.
>
> you can try but each openstack service (nova, neutorn, cinder, ...) is
> typically compised of several micocservices
> some like keystone and palcement are just rest apis in front of a db and
> dont use eventlet at all so are trivial to
> debug as a result. the rest have non tirival runtime interactions that
> cant be as eaislly decompsed.
> becuase of that nova and other project invest in our functional tests to
> allwo ues to emulate that multi process env
> and better reason about the code.
>
> the debugging functionality is documetned in the contibutor guide
>
> https://github.com/openstack/nova/blob/master/doc/source/contributor/develo…
> but we dont really adverstise that because none of the core maintainers
> use that any more and it has not been maintained
> for the better part of a decade. since we cant use this type of debugging
> with our ci system
> if we cant debug why a failure happens form the ci logs we add more logs
> because thyat will be useful for future us
> and operators in production. so our focus is on logging/monitoring for
> production rather then for developer eases
> simply because of the large techdebt that eventlet creates in this aready.
>
> i know nuetorn has had some better look with using debuggers then nova has,
>
> keystone/placmnet are eventlest free adn are just python web apps/rest
> apis so should "just work"
> in any python debugger.
>
> nova is prehaps the hardest to debug because of the legacy of the proejct
> so its not the best palce to start looking
> how this works. too be clear i have ran nova api, nova-conductor, nova
> scheduler and nova-compute all in 4 ide windows
> with 4 debuggers at once
> it was proably 3 year of working on openstack before i understood how to
> hack that ot work and since i learned
> how to properly debug without a debuggger, logs, code inspection,
> unit/fucntioal testing
> i have nver really need to do that again.
>
>
> gaining the ablity to run openstack in a debugger simply again, in a
> maintainable way is deffienly one of the reason im
> looking forward to removing eventlet but its still a lot of work.
> im not sure how helpful this was but this is proably as clsoe to a blog
> post as you will get.
>
> >
> > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote:
> >
> > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote:
> > > > How many more releases are you planning to include eventlet in
> > > > them? Seems they are removing any support for further development
> > > > according to their official website. So, is the community thinking
> > > > of any alternatives?
> > > [...]
> > >
> > > Just to expand on the first part of Sean's reply, most of what you
> > > want to know is probably answered here:
> > >
> > >
> https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html
> > >
> > > --
> > > Jeremy Stanley
> > >
>
>
1 year, 4 months
[nova][ptg] 2026.1 Gazpacho PTG summary
by Rene Ribaud
Hello everyone,
Last week was the PTG, thank you to all who joined and contributed to the
discussions!
I hope you enjoyed it and found it productive.
Again, this cycle we had a packed agenda with active participation
throughout the sessions.
For the Nova sessions (Tuesday to Friday), we had an average of 15 to 20
participants per session. Attendance was slightly higher during the
cross-project discussions, with an average of 25 to 30 people joining.
You can revisit all notes and discussions on the Etherpad here:
https://etherpad.opendev.org/p/r.9bac75a686ab0941572757b205c9415c
Below is a summary of the topics and outcomes from our PTG discussions.
I’ve tried to keep it as concise as possible, though we covered quite a lot
during the week.
**** Tuesday topics ****
#### 2025.2 Flamingo Retrospective ####
During the Flamingo cycle, 14 blueprints were accepted and 9 were fully
implemented, resulting in a 64% completion rate.
This is slightly lower than the 71% achieved during the previous Epoxy
cycle.
In addition, at least 26 bug fixes were merged during the cycle.
Review & Process Feedback
Eventlet Transition Plan, the migration continues to work as expected and
remains well-coordinated.
Release process was smooth with no major issues reported.
Recognition to Kamil for his work on the conductor, and to Sean, Dan, and
others for their consistent review efforts.
✅ Continue to prioritize the Eventlet removal as a key goal for the G cycle.
Branch Opening Policy, some uncertainty remains about what can be merged
between RC1 and GA, when master is technically open but still sensitive for
late merges.
Concerns were raised about backports and release stability.
✅ Allow merges on a case-by-case basis once the new master branch opens.
✅ Bauzas to propose a patch updating Nova contributor process
<https://docs.openstack.org/nova/latest/contributor/process.html>
<https://docs.openstack.org/nova/latest/contributor/process.html>
Bug Triage, the number of untriaged bugs remains high (around 180).
Efforts were noted, but time allocation and rotation remain challenging.
The end-of-meeting triage format was found ineffective for real-time work,
though still useful for discussing specific cases.
✅ Consider reviving a rotating triage roster if enough volunteers join.
✅ Keep triage discussions focused during the weekly meeting rather than
doing live triage.
Review Load, contributors noted increasing difficulty in finding time for
reviews.
Maintaining review velocity remains a cross-cycle challenge.
✅ Encourage feature-specific syncs or review sprints for high-priority
areas.
Feature Momentum, weekly syncs and structured updates help large features
maintain progress.
The VMware CI topic was cited as useful, though too many topics per meeting
could cause overload.
✅ Keep dedicated weekly or bi-weekly syncs for major features (e.g.
Eventlet removal).
✅ Limit meeting updates to 2-3 features per session.
Review Tracking, the previous Etherpad tracker was not effective.
However, having a central list remains important to avoid missing patches
and to monitor review health.
✅ Discuss an external or simplified review tracker during the next meeting
with more core reviewers.
✅ Keep the tracker short and up to date to maintain visibility.
#### 2025.2 Flamingo Planning ####
Hard spec freeze: December 4th
Milestone 2 (M2): January 8th
Feature Freeze (FF): February 26th
Release: End of March / early April.
The team discussed adjusting the timing of the spec freeze.
A soft spec freeze was proposed for November 20th, giving three weeks after
the PTG for contributors to submit specs.
Two spec review days will be scheduled between the soft and hard freezes.
Additionally, two implementation review days are planned: one around M2 for
early landings, and another roughly two weeks before M3.
✅ Adopt the proposed plan, while staying flexible on a case-by-case basis.
✅ Soft spec freeze on Nov 20th, followed by two spec review days before Dec
4th.
✅ Two implementation review days: one near M2 and one two weeks before M3.
✅ A +2 on a spec implies reviewers commit to reviewing the corresponding
implementation.
#### Upstream meeting schedule ####
✅ Move the weekly upstream meeting to Monday at 16:00 UTC.
✅ Keep the slot fixed (no alternating or skipping weeks).
✅ Uggla to update the meeting schedule and cycle accordingly.
#### Upstream bug triage follow up ####
The team discussed how to better organize bug triage follow-up for the
Gazpacho cycle.
A proposal was made to create a small group of volunteers who would receive
a few bugs to triage each week.
Each volunteer would handle up to three bugs, distributed by Uggla.
During the weekly upstream meeting, the team will check whether a dedicated
Q&A session is needed to discuss specific bugs.
If required, such a session will be scheduled every two weeks, ideally
right after the upstream meeting.
Otherwise, discussions will take place within the regular meeting.
✅ The team agreed to give this approach a try in the next cycle. The
overall sentiment seemed favorable, though details will likely need
adjustment as we go.
#### Eventlet removal ####
Significant progress was made since Flamingo.
In Flamingo, n-api, n-metadata, and n-sch could already be switched to
native threading via an environment variable, and the nova-next job runs
those services without Eventlet.
n-cond was nearly ready but missed feature freeze due to a late futurist
bug, which has since been fixed and released.
A py312-threading tox target and a Zuul job already validate a large part
of the unit tests without Eventlet.
In early Gazpacho, native threading for the conductor has successfully
landed.
Planned tasks for Gazpacho
-
Add native threading support for n-novncproxy and other proxies.
-
Add native threading support for n-compute.
-
Run all unit and functional tests in both threading and Eventlet modes,
keeping small exclude lists where needed.
-
Conduct scale testing using the fake driver, possibly extended to the
libvirt driver.
-
Switch n-api, n-metadata, and n-sch to native threading by default,
keeping Eventlet as fallback.
-
Introduce a nova-eventlet (legacy) CI job to ensure backward
compatibility.
-
Keep nova-next as the only job testing conductor, novncproxy, and
compute with threading until the H cycle.
-
Fix bugs reported after Eventlet removal.
-
After Eventlet removal, migrate from PyMySQL to libmysqlclient for
better DB performance.
✅ Kamil volunteered to handle the novncproxy migration.
✅ Start with nova-compute migration first to build confidence.
✅ Switch the default to threading as early as possible in Gazpacho.
✅ Add a dedicated Eventlet CI job to continue testing legacy mode.
✅ Use the hybrid plug job to test Eventlet mode even for services switched
to threading by default.
✅ Keep the work independent, but document the known issue with long-running
RPC calls.
#### Long-Running Sync REST API Calls ####
The team discussed how long-running synchronous REST API actions (like
volume or interface attach) can block nova-api threads once Eventlet is
removed.
With native threading, the limited number of threads makes blocking calls
more costly, unlike Eventlet which could spawn many lightweight greenlets.
This work is closely related but independent from the Eventlet removal
effort.
Dan started a spec on the topic, and it may be split into two (volume and
interface attach).
Detach operations are already async, and Manila shares follow a similar
model.
✅ Introduce a new microversion to keep backward compatibility.
✅ Convert volume and interface attach to async operations under that
microversion.
✅ Coordinate with Rajesh’s "show finish_time" (
https://review.opendev.org/q/topic:%22bp/show-instance-action-finish-time%22
) work to avoid race conditions.
✅ Dan and Johannes to update the spec.
✅ Further investigation is needed to determine if WSGI or API tuning could
help mitigate blocking.
#### vTPM Live Migration ####
The team reviewed how to handle TPM secret security policies during
instance operations.
Changing the assigned policy during resize is not supported, as it adds
complexity and can lead to image/flavor conflicts.
Rebuilds are already blocked for vTPM instances, so once a policy is set
via resize, it remains locked in.
Existing instances from previous releases are unaffected.
✅ Do not allow changing the TPM secret security policy after assignment.
✅ Remove the option to select the policy from the image for simplicity.
✅ Default policy is “user”, but compute nodes support all policies by
default.
✅ Document in the spec and release notes that deployers must define flavors
with hw:tpm_secret_security if they want to enable this.
✅ Mention that [libvirt]supported_tpm_secret_security = ['user', 'host',
'deployment'] can be adjusted by operators.
**** Wednesday topics ****
#### CPU Comparison During Live Migration ####
The team revisited the CPU compatibility check between source and
destination hosts during live migration.
If skip_cpu_compare_on_dest is enabled, some migrations may fail due to
missing pre-checks; if disabled, the check can sometimes be too strict.
The original goal is to avoid migrations to nodes running with QEMU
emulation, which would cause serious performance issues.
✅ Include getDomainCapabilities in the Nova pre-check to improve accuracy.
✅ Move the configuration flag from workaround to the libvirt section.
✅ Document potential unsafe edge cases so deployers can choose between the
safer (Nova) or more permissive (libvirt-only) approach.
### Start cross session with Cinder ###
#### NFS / NetApp NFS Online Volume Extend ####
The team discussed adding online volume extend support for NFS and NetApp
backends.
Instance operations (like shutdown or migration) should be blocked during
the extend process, and any new task_state must stay compatible with
existing microversions.
✅ Avoid using the event API, as it’s meant for fast operations.
✅ Introduce a dedicated API for this feature to manage versioning and
states.
✅ Consider making the Cinder call synchronous for better control.
✅ Konrad will update the spec in this way.
#### Hard-Coded io=native Setting for Cinder Backends ####
The team revisited the hard-coded io=native driver setting used for Cinder
volume backends (NFS, FC, and iSCSI).
This code dates back nearly ten years and has recently caused performance
issues and guest hangs in some environments.
The original reason for enforcing io=native is unclear.
✅ No one seems to know the reason for this original choice
✅ Move the configuration to a standard libvirt configuration option.
### End cross session with Cinder ###
#### Preserve NVRAM After Reboots and Migrations ####
The team discussed the proposal to preserve NVRAM data across instance
reboots and migrations.
The feature is ready for review, pending available reviewer bandwidth.
✅ Request a definitive answer from libvirt regarding file permission
handling through a bug report.
### Start cross session with Ironic/Cinder ###
A more complete summary of this joint session will be provided by the
Ironic team, as they can better capture the discussions and decisions made.
However, the Nova PTG notes still include a few relevant takeaways and
references for those interested.
### End cross session with Ironic/Cinder ###
### Start cross session with Manila ###
The teams discussed testing and integration plans between Nova and Manila.
Testing through Tempest was considered challenging due to configuration
complexity and potential circular dependencies.
Support for CephFS and NFS, hot attach, and live migration with newer OS
versions was identified as a goal.
✅ Manila will draft initial UX and workflow details to share with Nova for
review.
✅ Add a backlog spec in the Nova repository to track cross-team work.
✅ Manila contributors may help with open SDK and client patches as needed.
✅ Hot attach and live migration support are valuable goals but will need
explicit demand and prioritization to move forward.
### End cross session with Manila ###
### Start cross session with Neutron ###
A more complete summary of this joint session will be provided by the
Neutron team, as they can better capture the discussions and decisions made.
However, the Nova PTG notes still include a few relevant takeaways and
references for those interested.
### End cross session with Neutron ###
**** Thursday topics ****
#### Correct Firmware Detection for Stateless Firmware and AMD SEV/SEV-ES
####
The team discussed improving firmware detection for stateless firmware and
AMD SEV/SEV-ES.
Recent QEMU and libvirt versions now include firmware descriptor files that
expose these capabilities, allowing libvirt to automatically select the
right firmware without Nova duplicating that logic.
The group agreed that leveraging libvirt’s detection is the better
long-term approach.
To avoid regressions, Nova should continue to preserve existing firmware
for running instances during hard reboots, while new instances will rely on
libvirt’s detection from the start.
✅ Use libvirt’s firmware detection instead of maintaining custom logic in
Nova.
✅ Preserve the existing firmware for current instances on hard reboot. Let
libvirt select firmware for new ones.
✅ Takashi will write a spec describing the approach and detailing the
upgrade path.
✅ He will also check UEFI/stateless firmware tests in CI and run local
SEV/SEV-ES tests.
#### ROM-Type Firmware Support ####
The team continued the discussion on firmware selection, focusing on
ROM-type firmware used by AMD SEV-SNP, Intel TDX, and future Arm CCA.
Since recent libvirt versions can automatically select the appropriate
firmware and handle SMM when needed, Nova no longer needs to manage this
logic directly.
✅ Skip further work in Nova and rely on libvirt’s firmware auto-selection
mechanism.
#### AMD SEV-SNP Support ####
The team discussed initial plans for SEV-SNP support, which depends on
firmware and requires recent QEMU (≥9.2) and libvirt versions.
Work may be postponed to the 2026.2 cycle due to these dependencies.
For SEV-SNP, there is a need to provide hostData, an immutable and attested
field (up to 32 bytes) that can be set by Nova or external tools. Using
Nova metadata for this purpose was ruled out.
✅ Do not use Nova metadata to provide SEV-SNP host data.
✅ The instance POST API could be extended to allow providing this
information.
✅ Takashi will write an initial spec, keeping the design generic for both
SEV-SNP and TDX.
#### Generalize SEV / SEV-ES Code ####
The team reviewed the proposal to generalize the existing SEV/SEV-ES code
in preparation for future Confidential Computing architectures such as
Intel TDX and Arm CCA.
The work focuses on refactoring internal Nova code to make memory
encryption handling more generic, without changing any user-visible
behavior or API.
✅ Treat this as a code refactoring only (no external impact).
✅ Ensure it does not conflict with Takashi’s ongoing work.
✅ Implementation can proceed as a specless blueprint.
✅ No Tempest job available. Manual validation is acceptable, ideally with
screenshots or logs.
✅ Note: It’s possible to run Tempest manually using a flavor requesting
SEV/SEV-ES to verify correctness.
#### Arm CCA Support ####
The team discussed the roadmap for Arm CCA enablement in Nova.
Upstream dependencies include Linux Kernel (Dec 2025), QEMU (Feb 2026), and
libvirt (Mar 2026), with full availability expected around Ubuntu 26.04
(Apr 2026).
As a result, development and testing in OpenStack are expected to start
during the 2026.2 cycle.
✅ Features cannot merge until support is available in official
distributions (not custom kernels).
✅ CentOS Stream 10 may serve as an early test platform if it gains CCA
support before Ubuntu 26.04.
✅ Specs or PoCs can still be prepared in advance to ease future inclusion.
✅ Once libvirt support lands, the team will review the spec, targeting
early in the H (2026.2) cycle.
#### Show finish_time in Instance Action Details ####
The proposal adds a finish_time field to the instance action show API,
along with matching updates in the SDK and client.
Some review comments still need to be addressed before re-proposing the
patch for the G release.
✅ No objection to re-proposing the patch for Gazpacho (G).
✅ The existing spec remains valid.
✅ Gmaan will review previous comments to help move the implementation
forward.
### Start cross session with Glance ###
A more complete summary of this joint session will be provided by the
Glance team, as they can better capture the discussions and decisions made.
However, the Nova PTG notes still include a few relevant takeaways and
references for those interested.
### End cross session with Glance ###
#### Periodic Scheduler Update as Notification ####
The proposal suggests exposing Nova’s periodic resource tracker updates as
optional versioned notifications.
This would help services like Watcher obtain a more complete and up-to-date
view of compute resources, including NUMA topology and devices not yet
modeled in Placement.
While there are concerns about notification stability and data volume, the
team agreed it could be useful for cross-service integrations.
✅ Support the idea in principle.
✅ Be mindful of the volume of data included in notifications.
✅ Sean will prepare a spec proposal detailing the approach.
#### Resource Provider Weigher and Preferred/Avoided Traits ####
The proposal introduces a resource provider weigher to influence scheduling
decisions, with a longer-term goal of supporting preferred and avoided
traits.
This would allow external services, such as Watcher, to tag compute nodes
(e.g., with CPU or memory pressure) and guide the scheduler to favor or
avoid specific hosts.
✅ There are concerns about validating the weigher’s behavior at scale,
ensuring it works correctly with a large number of instances.
✅ Continue developing this approach.
✅ Improve the test framework, despite existing challenges.
✅ Add Monte Carlo–style functional tests to validate behavior in complex,
non-symmetric scenarios.
#### Proxying OVH’s Monitoring-Aware Scheduling ####
The team discussed OVH’s proposal to enhance scheduling decisions by
integrating external monitoring data.
The idea is to add a filter or weigher that delegates its logic to an
external plugin, allowing operators to plug in custom metrics (e.g.,
Prometheus, InfluxDB) without modifying Nova directly.
While the use case is relevant, existing out-of-tree filters and weighers
already allow similar integrations.
One of the main challenges identified was how to share and maintain such
code consistently across the community.
✅ Current Nova interfaces likely already support this use case.
✅ Proposal to create a dedicated repository on OpenDev for sharing and
maintaining custom filters and weighers.
**** Friday topics ****
#### LY Corp – Graceful Shutdown Proposal ####
The team discussed LY Corp’s proposal to implement graceful shutdown
handling in Nova.
The goal is to stop accepting new RPC requests while allowing in-progress
operations, such as instance boots or live migrations to complete safely.
Different designs were compared, including stopping RPC listeners or using
the upcoming oslo.messaging HTTP driver, which can handle clean shutdowns
more reliably.
The consensus was that the complete solution must involve oslo.messaging,
allowing Nova services to dynamically subscribe or unsubscribe from topics.
This approach also implies an upgrade impact, requiring versioned changes
to compute and scheduler topics.
✅ The current proposal needs further work, as live migration cases were not
yet considered.
✅ Gmaan will continue investigation and refine the approach based on
oslo.messaging improvements.
✅ A revised spec will be submitted once ready.
✅ Gmaan confirmed commitment to follow up and lead this feature.
#### Offline VM Migration Using HTTP-Based File Server ####
The team discussed a proposal to enable offline VM migration using an
HTTP-based file server.
The preferred approach is to use WebDAV, which provides a standard protocol
and existing server implementations.
Nova itself will not host the web server and will be restricted to paths
under /var/lib/nova.
✅ Use WebDAV as the preferred protocol.
✅ Avoid introducing non-standard protocols in Nova.
✅ Nova will not spawn or host the web server directly.
✅ Define a URL template in configuration for the remote filesystem driver
(e.g., remotefs_url = https://%(host)s:9999/path/to/dav).
✅ Add DevStack support to test the feature in CI.
#### PCI and VGPU Scheduling Performance in Placement
The team revisited the performance issues with PCI and VGPU scheduling in
Placement.
Two workaround sets have already been implemented to reduce the number of
valid and invalid candidates, but the core scalability problem of the GET
/allocation_candidates query remains, especially in large, symmetric
provider trees.
A long-term constraint solver approach is under consideration, and the
discussion will continue once more data is available from current
deployments.
✅ Keep the current workarounds active and gather feedback over the next six
months.
✅ Nova-next may run with these settings enabled to collect early results.
✅ Revisit the topic and defaulting decisions at the next PTG.
✅ Keep the issue open for urgent improvements if needed before the next
cycle.
#### Multithread and Multiqueue Support for Libvirt ####
The team discussed adding multithread and multiqueue support in Nova for
libvirt-based instances.
While QEMU already enables multiqueue for virtio-blk and virtio-scsi by
default, the main focus is now on IOThread support and improving
parallelism during I/O and live migrations.
✅ Enable IOThreads with one thread per device by default (handled as a
specless blueprint).
✅ virtio-blk and virtio-scsi specs are parked.
✅ Cancel the multiqueue work, since QEMU now manages it natively.
✅ The parallel migration blueprint is approved as specless.
✅ The earlier proposed spec will be abandoned in favor of this simplified
path forward.
#### AZ-Aware Scheduling (Soft Anti-Affinity) ####
The team discussed improving availability zone–aware scheduling by
introducing a soft anti-affinity policy for server groups.
✅ The overall direction is good, but the spec still needs more work.
✅ The team is broadly supportive of continuing in this direction.
✅ A spec will be proposed by Dmitriy.
#### NUMA-Aware Placement (CloudFerro Proposal) ####
The team discussed CloudFerro’s proposal to enable NUMA-aware scheduling in
Placement, based on the old Ussuri spec.
The goal is to represent NUMA nodes as resource providers to allow better
locality handling for CPU and memory resources.
However, this raises potential performance concerns, especially when
combined with PCI in Placement, as both increase candidate permutations.
✅ Ensure NUMA and PCI in Placement work together correctly before expanding
scope.
✅ Do not move PCI devices under NUMA subtrees yet.
✅ Add functional tests to validate the combined behavior.
✅ Consider extending CI to run on hosts with real NUMA topology.
✅ Enable NUMA awareness in nova-next (single NUMA node for now).
✅ Investigate with the infra team the possibility of adding dual-NUMA test
nodes.
✅ Improvements to Placement’s group_policy remain out of scope for this
work.
✅ The team is happy with the current direction and looks forward to future
patches landing in master.
#### WSGI Script Removal ####
The team discussed removing the legacy wsgi_script support from Nova.
No objections were raised, and no spec is required.
✅ Proceed with the removal, the team is aligned and ready to move forward.
#### Improving Cyborg Support ####
The team discussed reviving Cyborg integration efforts in Nova.
Several aspects remain unfinished, including proper mdev GPU XML
generation, support for additional device buses (block, NVMe, CXL, USB),
and handling ownership overlap between Nova and Cyborg in Placement.
The plan is to resume this work gradually, starting with the most mature
components.
✅ Restart the Cyborg–Nova integration effort.
✅ Split the work into multiple specs, as each part is complex enough to
warrant separate discussion.
✅ Focus first on mdev support, then extend to other buses (block, NVMe,
etc.).
✅ Investigate Placement ownership to avoid inventory conflicts between Nova
and Cyborg.
If you've read this far, thank you! 🙏
If you spot any mistakes or missing points, please don't hesitate to let me
know.
Best regards.
René.
1 month, 4 weeks
Re: [eventlet-removal]When to drop eventlet support
by Dmitriy Rabotyagov
In case you try to use a 32gb box with 16 cores as a controller for
OpenStack - it will blow off with default amount of workers for wsgi and
/or eventlet apps.
While you can argue this should not be used as production setup, this can
be totally valid for sandboxes and we wanna provide consistent and reliable
behavior for users.
But my argument was not in if/how we want to fine-tune deployments, but
also understand and provide means to define what's needed as well as
potential ability to revert in worst case scenario as a temporary
workaround.
So still some variables and logic would be introduced from what I
understand today.
On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote:
>
> On 16/06/2025 13:27, Dmitriy Rabotyagov wrote:
> >
> >
> >
> > sayint its FUD is not helpful.
> >
> > we got a driect ask form operator and soem core to not do a hard
> > switch
> > over.
> >
> > and while i wanted to only support one model for each binary at a
> > time
> > we were sepcificly ask to make it configurable.
> >
> > > In the later case, your only available action is help fixing
> > bugs. It
> > > is not up to the operators to double-guess what may or may not
> > work.
> >
> > correct we are not planning to document how to change mode we were
> > planning to only use this configuration in ci and operator would be
> >
> >
> > Well, we'd need to have that communicated so that deployment toolings
> > could adopt their setup to changes, as, for instance, in OSA amount of
> > eventlet workers are calculated based on the system facts, so we'd
> > need to change the logic and also suggest how users should treat this
> > new logic for their systems.
>
> why is OSA doing that at all today?
> we generally don't recommend changing those values from the default
> unless you really know what your doing.
> i don't think other installer do that.
> tripleo, kolla-ansbile and our new golang based installer do not, nor
> does devstack so its surprising to me that OSA would change such low
> level values
> by default.
>
> we will document any new config options we and and we are documentation
> how to tune the new options for thread pools but we do not expect
> installation
> tools to modify them by default. we are explicitly not making the
> options based on the amount of resources on the host i.e. dynamically
> calculated based
> on the number of CPU cores.
>
> for example we are explicitly setting the number of scatter_gather
> thread in the the dedicated thread pool to 5
> why its a nice small number that will work for most people out of the box.
>
> can you adjust it, yes but it scale based on the number of nova cells
> you have an 99% wont have more then 5 cells.
>
> using information about the host where the API is deployed to infer the
> value of that would be incorrect.
>
> you can really only make an informed decision about how to tune that
> based on monitoring the usage of the pool.
>
> that how we expect most of the other tuning options to go as well.
>
> our defaults in nova tend to be higher then you would actually need in a
> real environment so while it may make sense to reduce
> them we try to make sure the work out of the box for most people.
>
> gibi id building up
>
> https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co…
>
> as part of nova move to encode this but our goal is that deployment
> tools shoudl not need to be modifyed to tune these
> valued by defualt.
>
> >
> > So it will be kinda documented in a way after all.
> >
> >
> > told for a given release deploy this way.
> >
> > this is an internal impelation detail however we are not prepared to
> > deprecate usign eventlet until we are convicned
> >
> > that we can run properly without it.
> >
> > > For beginners, this would be a horrible nightmare if default
> > options
> > > simply wouldn't work. We *must* ship OpenStack working by default.
> > no one is suggesting we do otherwise.
> > >
> > > Cheers,
> > >
> > > Thomas Goirand (zigo)
> > >
> >
>
>
6 months, 2 weeks
Re: Eventlet and debugging
by engineer2024
Appreciate taking time to post this. I will look into the options and try
them... Have a nice week ahead
On Thu, 15 Aug 2024, 03:53 , <smooney(a)redhat.com> wrote:
> On Wed, 2024-08-14 at 23:06 +0100, smooney(a)redhat.com wrote:
> > On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote:
> > > Thanks to both of you for the response . I try to insert pdb
> breakpoint at
> > > some point in the code files and start the service. Then I issue a nova
> > > command to check how the flow works. From then on I use the pdb
> commands to
> > > check the function calls. This to me gives a very convenient way of
> > > learning the code connections, since I don't have an idea of how these
> > > functions and modules r related. U people as developers have a design
> to
> > > start with and work on from there. So u may know the purpose of each
> > > function, class, etc.
> > >
> > > Since nova api doesn't support pdb becos of eventlet design, I learned
> to
> > > print those objects and find their information.
> > >
> > > But with pdb, it's a smooth flow to inspect the class attributes, their
> > > functions, arguments, types etc. I can even test those objects the
> python
> > > way since pdb supports python prompt as well. Of course I do all this
> on
> > > dev systems though...
> >
> > this is going to be a long responce and its a bit rambely but im goign
> to provide
> > you with the context and poitn er try and figure out how to do this for
> your self.
> >
> > first thing to note is while its possibel it not trivial and the payoff
> is proably not
> > worth it in the long run. we do not have up to day docs for how to do
> this because
> > this is not how peole learn, contibute to ro develop nova in genral.
> >
> >
> > nova does have a remote debug facilitate that is sepreate for the
> eventlet backdor to enable
> > you to use an ide to debug remotely.
> > that was added around 2012 and has worked on an off to vering degrees
> since then.
> >
> > the first thing to understand about nova and several other project is
> that they are a distibuted system
> > i.e. the applciaiton that compise a given service like nova run in
> multiple service which are interconnected
> > vai a message bus with in a service and rest apis are exposed for inter
> service comunication.
> >
> > when you enable a debugger and stop on a breakpoitn only one of the
> process is stop and the other contiue
> > to exsculte which means that RPC calls, heathbeats and other asynconos
> tasks can and will timeout and fail.
> >
> > you cannot debug a distibuted system in teh same way as you woudl a
> simple applction where all state
> > is executed in a signle main loop.
> >
> > the act of observing the internal behavior of the nova api or
> nova-comptue agent will change its behavior.
> >
> > nova has suppoort for remote debugging that was added about 12 years ago
> > https://wiki.openstack.org/wiki/Nova/RemoteDebugging
> > https://github.com/openstack/nova/blob/master/nova/debugger.py
> >
> > with that supprot you can use an idea like pycharm to connect to the
> remote debugger and attempt to
> > debug a process.
> >
> > im currently propsoing removing that as part of the removal of eventlet
> partly because it shoudl not be needed
> > when we dont have enventlet and mainly because it didnt work very well
> when i last used it.
> >
> > some ides have supprot fo gevent
> >
> > gevent is a fork of eventlet which was forked form some of hte libiares
> in stackless python
> > eventlet converts a single thread python interperater in a cooperative
> concurent multi userland process.
> >
> by the way i forgot to say if you want to debug eventlet mokey patch code
> adn your using a debugger that supprot gevent it mostly work with eventlet
> too if you enabel the gevnet
> supprot in pycham which is disabeld by defaut.
>
> i have had it workk reallly well for months and then not work for years
> and suddenly start workign agian.
> so the proablme with usign a debugger with nova is not getting it working
> initally its keepign it working
> and setting it up on a new machine if you create a new devstack vm for dev
> again.
>
> its not hard, its hard to do repatbly, so blogs, docs and even automation
> tends to bit wroth quickly.
>
> > what does that mean , each python function is not operating in a
> seperate os stackframe, it has heap allcoated
> > stack frames that allow context swtichs between userland thread at any
> tiem a funnction woudl do blocking io
> >
> > the reason single step debugging does not work well with gdb is that
> when you single step the eventlet engine
> > is often going to end up context switching into the midel of another
> function.
> >
> > to work around this you just set breakpoint at the line you want to go
> to next and use continue instead fo single step
> >
> > one of the rules when using eventlet is that if your going to monkey
> patch you need to monkey patch everyting early
> >
> > in addtion too the remote debuging facilites we can disabl mokey
> patching entirly.
> > you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1
> > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87
> >
> > this will entirly break nova-compute and several other nova process as
> they have
> > effectivly infinity loops to pool for rpc messags, monitor hyperviors
> ectra.
> >
> > if the remote debugger is used it disables patching the thread lib
> >
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46
> >
> > that is prartly why usign the remote debugger is kind fo buggy in
> itsself.
> > the code is not inteded to work that way.
> >
> > these fucntionlatiy mainly exist to allow debuging tests not the
> actulaly running code.
> >
> > if i need to debug test i normaly jsut comment out
> >
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89
> > and run the test in my ides(pycharm,vscodes) test debbuger.
> > i normally code in nano/emacs but is do use ides if i really cant figure
> out what happening
> > however i genreally start with just adding log lines.
> >
> > when i first started workign on opentack in 2013 i very much liked using
> an ide and debugger
> > and i woudl often spend time trying to get nova to work in a debugger.
> >
> > over tiem i generally found that that was not worth the effort.
> > keepign a devstack deployment working that way was often more work then
> doing print staement
> > or better writing functional or unit test to repoduce the problem.
> >
> > you previous mails ask about debuging nova's api.
> >
> > if you want to do that its one of the easiest thigns to debug.
> > first do _not_ use the wsgi script
> > adding apache of modwsgi is just going to make your life harder.
> >
> > just disable eventlet monkey patching, by commetiing or usign the env
> varible
> > and run the nova-api console script directly under pdb or your debugger
> of choice.
> > that will allow you to basiclly single step debug without any issues as
> it will not be using eventlet
> > anymore and it has not infinitly loops or perodic tasks so it mostly
> just works when not using eventlet.
> >
> > that becase it didnt use eventlet at all in the api until about 2016 and
> even today it only really uses it in one
> > place
> > in my eventlet removal serise i split nova in to the part that use
> eventlet directly and the parts that dont in the
> > second path
> >
> > https://review.opendev.org/c/openstack/nova/+/904424
> >
> > anything under nova/cmd/eventlet/ needs eventlet today
> > everything under nova/cmd/standalone/ does not.
> >
> > the approch of doign an api call and following it in a debugger is not a
> good way in my expericne to learn how
> > somethign
> > like nova really works. it can hellp to a limtied degree but you cant
> eaislly debug across multipel processes.
> > a minium nova deployment has 4 process typically more.
> >
> > you can try but each openstack service (nova, neutorn, cinder, ...) is
> typically compised of several micocservices
> > some like keystone and palcement are just rest apis in front of a db and
> dont use eventlet at all so are trivial to
> > debug as a result. the rest have non tirival runtime interactions that
> cant be as eaislly decompsed.
> > becuase of that nova and other project invest in our functional tests to
> allwo ues to emulate that multi process env
> > and better reason about the code.
> >
> > the debugging functionality is documetned in the contibutor guide
> >
> https://github.com/openstack/nova/blob/master/doc/source/contributor/develo…
> > but we dont really adverstise that because none of the core maintainers
> use that any more and it has not been
> > maintained
> > for the better part of a decade. since we cant use this type of
> debugging with our ci system
> > if we cant debug why a failure happens form the ci logs we add more logs
> because thyat will be useful for future us
> > and operators in production. so our focus is on logging/monitoring for
> production rather then for developer eases
> > simply because of the large techdebt that eventlet creates in this
> aready.
> >
> > i know nuetorn has had some better look with using debuggers then nova
> has,
> >
> > keystone/placmnet are eventlest free adn are just python web apps/rest
> apis so should "just work"
> > in any python debugger.
> >
> > nova is prehaps the hardest to debug because of the legacy of the
> proejct so its not the best palce to start looking
> > how this works. too be clear i have ran nova api, nova-conductor, nova
> scheduler and nova-compute all in 4 ide windows
> > with 4 debuggers at once
> > it was proably 3 year of working on openstack before i understood how to
> hack that ot work and since i learned
> > how to properly debug without a debuggger, logs, code inspection,
> unit/fucntioal testing
> > i have nver really need to do that again.
> >
> >
> > gaining the ablity to run openstack in a debugger simply again, in a
> maintainable way is deffienly one of the reason
> > im
> > looking forward to removing eventlet but its still a lot of work.
> > im not sure how helpful this was but this is proably as clsoe to a blog
> post as you will get.
> >
> > >
> > > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote:
> > >
> > > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote:
> > > > > How many more releases are you planning to include eventlet in
> > > > > them? Seems they are removing any support for further development
> > > > > according to their official website. So, is the community thinking
> > > > > of any alternatives?
> > > > [...]
> > > >
> > > > Just to expand on the first part of Sean's reply, most of what you
> > > > want to know is probably answered here:
> > > >
> > > >
> https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html
> > > >
> > > > --
> > > > Jeremy Stanley
> > > >
> >
>
>
1 year, 4 months
[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-9)
by Goutham Pacha Ravi
Hello Stackers,
We're counting down 9 weeks to the 2024.2 (Dalmatian) release [1].
OpenStack projects that follow the cycle-with-rc release model [2]
will have their code freeze in 4 weeks. Between now and the release,
the release team will communicate with us weekly to coordinate
deliverables across the community. The elections for OpenStack PTLs
and Technical Committee members are around the corner as well, so now
is a good time to ensure that you are eligible to vote in these
elections! Please navigate to:
https://civs1.civs.us/cgi-bin/opt_in.pl
Enter your preferred email address on Gerrit and request activation.
It only takes a minute! :)
The following are updates from the OpenStack Technical Committee
during the past week.
=== Weekly Meeting ===
The TC had a productive meeting via IRC on OFTC's #openstack-tc
channel last week [3]. We discussed follow-ups for the eventlet
removal goal proposal [4] and recent updates to inactive and emerging
OpenStack projects. We spent some time assessing gate health concerns,
including an issue with a change to `setuptools` that requires us to
bump the upper constraint on the "packaging" library in stable/2023.1
[5]. This was important to get some CVE fixes landed in various
OpenStack code repositories. Finally, we refreshed our understanding
of the TC's house rules. To summarize:
- Changes to the TC charter require at least 2/3 of the TC voting in
favor, while governance motions require at least 1/3 of the TC to vote
in favor.
- TC members can vote positively, negatively, or abstain (using the
“RollCall-Vote” in Gerrit).
- Decisions need more positive votes than negative votes (ties mean
the proposal is rejected).
- Any motion or charter-change is deliberately open at least a week
after it was last proposed/updated, allowing the TC and the wider
community to participate fully.
- Significant governance changes take a while to merge because of the
importance of ensuring full TC and community engagement in votes.
Our next weekly meeting is tomorrow (2024-07-30) at 1800 UTC via IRC
on OFTC's #openstack-tc channel. You can find the agenda on the
OpenStack TC wiki page [6]. Please consider joining us and remember,
you can edit the wiki and propose new topics!
=== Governance Proposals ===
==== Merged ====
- Add `os-test-images` under glance |
https://review.opendev.org/c/openstack/governance/+/925044
==== Open for review/discussion ====
- [Follow-up] Fix typos in the eventlet proposal |
https://review.opendev.org/c/openstack/governance/+/924944
- Update criteria for inactive projects to become active again |
https://review.opendev.org/c/openstack/governance/+/921500
- Follow up on the affiliation diversity handling |
https://review.opendev.org/c/openstack/governance/+/923876
- Inactive state extensions: Monasca |
https://review.opendev.org/c/openstack/governance/+/923919
=== Upcoming Events ===
- 2024-08-12: Call for Proposals due for the OpenInfra Days NA [7]
- 2024-08-14: Nominations open for 2025.1 OpenStack PTL+TC elections
- 2024-08-24: OpenInfra Days, Vietnam: https://2024.vietopeninfra.org/
- 2024-08-29: Dalmatian-3 milestone
- 2024-09-03: OpenInfra Summit Asia, Suwon
The OpenStack Technical Committee serves you! We're always willing to
chat and would love feedback:
=== How to contact the TC ===
You can reach out to the TC in multiple ways:
1. Email: Send an email with the tag [tc] on this email list.
2. Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel.
3. Join us at our weekly meeting: The Technical Committee meets every
week on Tuesdays at 1800 UTC [7].
Thank you very much for reading!
On behalf of the OpenStack TC,
Goutham Pacha Ravi (gouthamr)
OpenStack TC Chair
[1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html
[2] OpenStack cycle-with-rc model:
https://releases.openstack.org/reference/release_models.html#cycle-with-rc
[3] TC Meeting IRC Logs, 2024-07-23:
https://meetings.opendev.org/meetings/tc/2024/tc.2024-07-23-18.00.log.html
[4] Remove Eventlet From Openstack:
https://review.opendev.org/c/openstack/governance/+/902585
[5] Update packaging constraint in stable/2023.1:
https://review.opendev.org/c/openstack/requirements/+/924764
[6] TC Meeting, 2024-07-30:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting
[7] Call for Proposals, OpenInfra Days NA:
https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc…
1 year, 5 months
Re: Eventlet and debugging
by engineer2024
Actually, when I was in my graduate/post graduate studies in computer
science, I used to think that whoever is technically smart but unable to
communicate in english would find a hard time in the industry.
But after starting my career in IT and seeing many brilliant minds, I am
convinced that language is not as important as the skill itself. We just
need to understand each other in anyway possible, not only
in english language. So many of my colleagues have gone on to higher
positions in IT now despite their lack of proper english communication. So
I think if you are able to express your ideas in any form, thats enough.
Yes its important for certain roles like sales , client/customer facing.
But not so for engineers.
And regarding Sean's typos, I can understand he is quick typing on a mobile
device or so, that these catch up. No issues though...
On Thu, Aug 15, 2024 at 1:06 PM Mike Carden <mike.carden(a)gmail.com> wrote:
> In almost a decade of reading smooney(a)redhat.com's Lysdexic posts, this
> is the longest one I have ever read. No criticism intended at all - none
> whatsoever.
>
> BUT... I am still absolutely Gobsmacked that anyone can possibly write
> even vaguely working code, if their written English communications are so
> comprehensively broken. How does anyone get away with ignoring Spelling and
> Grammar in Code?
>
> I love your work Sean, but I do struggle to understand almost everything
> you type.
>
> --
> MC
>
>
>
> On Thu, 15 Aug 2024 at 08:07, <smooney(a)redhat.com> wrote:
>
>> On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote:
>> > Thanks to both of you for the response . I try to insert pdb breakpoint
>> at
>> > some point in the code files and start the service. Then I issue a nova
>> > command to check how the flow works. From then on I use the pdb
>> commands to
>> > check the function calls. This to me gives a very convenient way of
>> > learning the code connections, since I don't have an idea of how these
>> > functions and modules r related. U people as developers have a design
>> to
>> > start with and work on from there. So u may know the purpose of each
>> > function, class, etc.
>> >
>> > Since nova api doesn't support pdb becos of eventlet design, I learned
>> to
>> > print those objects and find their information.
>> >
>> > But with pdb, it's a smooth flow to inspect the class attributes, their
>> > functions, arguments, types etc. I can even test those objects the
>> python
>> > way since pdb supports python prompt as well. Of course I do all this on
>> > dev systems though...
>>
>> this is going to be a long responce and its a bit rambely but im goign to
>> provide
>> you with the context and poitn er try and figure out how to do this for
>> your self.
>>
>> first thing to note is while its possibel it not trivial and the payoff
>> is proably not
>> worth it in the long run. we do not have up to day docs for how to do
>> this because
>> this is not how peole learn, contibute to ro develop nova in genral.
>>
>>
>> nova does have a remote debug facilitate that is sepreate for the
>> eventlet backdor to enable
>> you to use an ide to debug remotely.
>> that was added around 2012 and has worked on an off to vering degrees
>> since then.
>>
>> the first thing to understand about nova and several other project is
>> that they are a distibuted system
>> i.e. the applciaiton that compise a given service like nova run in
>> multiple service which are interconnected
>> vai a message bus with in a service and rest apis are exposed for inter
>> service comunication.
>>
>> when you enable a debugger and stop on a breakpoitn only one of the
>> process is stop and the other contiue
>> to exsculte which means that RPC calls, heathbeats and other asynconos
>> tasks can and will timeout and fail.
>>
>> you cannot debug a distibuted system in teh same way as you woudl a
>> simple applction where all state
>> is executed in a signle main loop.
>>
>> the act of observing the internal behavior of the nova api or
>> nova-comptue agent will change its behavior.
>>
>> nova has suppoort for remote debugging that was added about 12 years ago
>> https://wiki.openstack.org/wiki/Nova/RemoteDebugging
>> https://github.com/openstack/nova/blob/master/nova/debugger.py
>>
>> with that supprot you can use an idea like pycharm to connect to the
>> remote debugger and attempt to
>> debug a process.
>>
>> im currently propsoing removing that as part of the removal of eventlet
>> partly because it shoudl not be needed
>> when we dont have enventlet and mainly because it didnt work very well
>> when i last used it.
>>
>> some ides have supprot fo gevent
>>
>> gevent is a fork of eventlet which was forked form some of hte libiares
>> in stackless python
>> eventlet converts a single thread python interperater in a cooperative
>> concurent multi userland process.
>>
>> what does that mean , each python function is not operating in a seperate
>> os stackframe, it has heap allcoated
>> stack frames that allow context swtichs between userland thread at any
>> tiem a funnction woudl do blocking io
>>
>> the reason single step debugging does not work well with gdb is that when
>> you single step the eventlet engine
>> is often going to end up context switching into the midel of another
>> function.
>>
>> to work around this you just set breakpoint at the line you want to go to
>> next and use continue instead fo single step
>>
>> one of the rules when using eventlet is that if your going to monkey
>> patch you need to monkey patch everyting early
>>
>> in addtion too the remote debuging facilites we can disabl mokey patching
>> entirly.
>> you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1
>> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87
>>
>> this will entirly break nova-compute and several other nova process as
>> they have
>> effectivly infinity loops to pool for rpc messags, monitor hyperviors
>> ectra.
>>
>> if the remote debugger is used it disables patching the thread lib
>> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46
>>
>> that is prartly why usign the remote debugger is kind fo buggy in itsself.
>> the code is not inteded to work that way.
>>
>> these fucntionlatiy mainly exist to allow debuging tests not the actulaly
>> running code.
>>
>> if i need to debug test i normaly jsut comment out
>> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89
>> and run the test in my ides(pycharm,vscodes) test debbuger.
>> i normally code in nano/emacs but is do use ides if i really cant figure
>> out what happening
>> however i genreally start with just adding log lines.
>>
>> when i first started workign on opentack in 2013 i very much liked using
>> an ide and debugger
>> and i woudl often spend time trying to get nova to work in a debugger.
>>
>> over tiem i generally found that that was not worth the effort.
>> keepign a devstack deployment working that way was often more work then
>> doing print staement
>> or better writing functional or unit test to repoduce the problem.
>>
>> you previous mails ask about debuging nova's api.
>>
>> if you want to do that its one of the easiest thigns to debug.
>> first do _not_ use the wsgi script
>> adding apache of modwsgi is just going to make your life harder.
>>
>> just disable eventlet monkey patching, by commetiing or usign the env
>> varible
>> and run the nova-api console script directly under pdb or your debugger
>> of choice.
>> that will allow you to basiclly single step debug without any issues as
>> it will not be using eventlet
>> anymore and it has not infinitly loops or perodic tasks so it mostly just
>> works when not using eventlet.
>>
>> that becase it didnt use eventlet at all in the api until about 2016 and
>> even today it only really uses it in one place
>> in my eventlet removal serise i split nova in to the part that use
>> eventlet directly and the parts that dont in the
>> second path
>>
>> https://review.opendev.org/c/openstack/nova/+/904424
>>
>> anything under nova/cmd/eventlet/ needs eventlet today
>> everything under nova/cmd/standalone/ does not.
>>
>> the approch of doign an api call and following it in a debugger is not a
>> good way in my expericne to learn how somethign
>> like nova really works. it can hellp to a limtied degree but you cant
>> eaislly debug across multipel processes.
>> a minium nova deployment has 4 process typically more.
>>
>> you can try but each openstack service (nova, neutorn, cinder, ...) is
>> typically compised of several micocservices
>> some like keystone and palcement are just rest apis in front of a db and
>> dont use eventlet at all so are trivial to
>> debug as a result. the rest have non tirival runtime interactions that
>> cant be as eaislly decompsed.
>> becuase of that nova and other project invest in our functional tests to
>> allwo ues to emulate that multi process env
>> and better reason about the code.
>>
>> the debugging functionality is documetned in the contibutor guide
>>
>> https://github.com/openstack/nova/blob/master/doc/source/contributor/develo…
>> but we dont really adverstise that because none of the core maintainers
>> use that any more and it has not been maintained
>> for the better part of a decade. since we cant use this type of debugging
>> with our ci system
>> if we cant debug why a failure happens form the ci logs we add more logs
>> because thyat will be useful for future us
>> and operators in production. so our focus is on logging/monitoring for
>> production rather then for developer eases
>> simply because of the large techdebt that eventlet creates in this aready.
>>
>> i know nuetorn has had some better look with using debuggers then nova
>> has,
>>
>> keystone/placmnet are eventlest free adn are just python web apps/rest
>> apis so should "just work"
>> in any python debugger.
>>
>> nova is prehaps the hardest to debug because of the legacy of the proejct
>> so its not the best palce to start looking
>> how this works. too be clear i have ran nova api, nova-conductor, nova
>> scheduler and nova-compute all in 4 ide windows
>> with 4 debuggers at once
>> it was proably 3 year of working on openstack before i understood how to
>> hack that ot work and since i learned
>> how to properly debug without a debuggger, logs, code inspection,
>> unit/fucntioal testing
>> i have nver really need to do that again.
>>
>>
>> gaining the ablity to run openstack in a debugger simply again, in a
>> maintainable way is deffienly one of the reason im
>> looking forward to removing eventlet but its still a lot of work.
>> im not sure how helpful this was but this is proably as clsoe to a blog
>> post as you will get.
>>
>> >
>> > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote:
>> >
>> > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote:
>> > > > How many more releases are you planning to include eventlet in
>> > > > them? Seems they are removing any support for further development
>> > > > according to their official website. So, is the community thinking
>> > > > of any alternatives?
>> > > [...]
>> > >
>> > > Just to expand on the first part of Sean's reply, most of what you
>> > > want to know is probably answered here:
>> > >
>> > >
>> https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html
>> > >
>> > > --
>> > > Jeremy Stanley
>> > >
>>
>>
1 year, 4 months
Re: [eventlet-removal]When to drop eventlet support
by Sean Mooney
On 16/06/2025 15:33, Dmitriy Rabotyagov wrote:
> Also let's keep in mind, that only nova (with placement) will spawn 64
> threads on such setup by default.
yep see my other reply.
you mixign up workers and eventlet threads.
those are two very diffent things.
we likely shoudl change the upstream default for workers to 1.
we do tend to override that by defualt in installer tools.
>
> And then all really depends on set of services to launch on such setup.
>
> So from deployment tooling protective you have all required data to
> rollout not instantly oom-ing setup at cost of amount of request
> services can process in parallel.
>
> On Mon, 16 Jun 2025, 15:24 Dmitriy Rabotyagov,
> <noonedeadpunk(a)gmail.com> wrote:
>
> In case you try to use a 32gb box with 16 cores as a controller
> for OpenStack - it will blow off with default amount of workers
> for wsgi and /or eventlet apps.
>
> While you can argue this should not be used as production setup,
> this can be totally valid for sandboxes and we wanna provide
> consistent and reliable behavior for users.
>
> But my argument was not in if/how we want to fine-tune
> deployments, but also understand and provide means to define
> what's needed as well as potential ability to revert in worst case
> scenario as a temporary workaround.
> So still some variables and logic would be introduced from what I
> understand today.
>
>
> On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote:
>
>
> On 16/06/2025 13:27, Dmitriy Rabotyagov wrote:
> >
> >
> >
> > sayint its FUD is not helpful.
> >
> > we got a driect ask form operator and soem core to not
> do a hard
> > switch
> > over.
> >
> > and while i wanted to only support one model for each
> binary at a
> > time
> > we were sepcificly ask to make it configurable.
> >
> > > In the later case, your only available action is help
> fixing
> > bugs. It
> > > is not up to the operators to double-guess what may or
> may not
> > work.
> >
> > correct we are not planning to document how to change
> mode we were
> > planning to only use this configuration in ci and
> operator would be
> >
> >
> > Well, we'd need to have that communicated so that deployment
> toolings
> > could adopt their setup to changes, as, for instance, in OSA
> amount of
> > eventlet workers are calculated based on the system facts,
> so we'd
> > need to change the logic and also suggest how users should
> treat this
> > new logic for their systems.
>
> why is OSA doing that at all today?
> we generally don't recommend changing those values from the
> default
> unless you really know what your doing.
> i don't think other installer do that.
> tripleo, kolla-ansbile and our new golang based installer do
> not, nor
> does devstack so its surprising to me that OSA would change
> such low
> level values
> by default.
>
> we will document any new config options we and and we are
> documentation
> how to tune the new options for thread pools but we do not expect
> installation
> tools to modify them by default. we are explicitly not making the
> options based on the amount of resources on the host i.e.
> dynamically
> calculated based
> on the number of CPU cores.
>
> for example we are explicitly setting the number of
> scatter_gather
> thread in the the dedicated thread pool to 5
> why its a nice small number that will work for most people out
> of the box.
>
> can you adjust it, yes but it scale based on the number of
> nova cells
> you have an 99% wont have more then 5 cells.
>
> using information about the host where the API is deployed to
> infer the
> value of that would be incorrect.
>
> you can really only make an informed decision about how to
> tune that
> based on monitoring the usage of the pool.
>
> that how we expect most of the other tuning options to go as well.
>
> our defaults in nova tend to be higher then you would actually
> need in a
> real environment so while it may make sense to reduce
> them we try to make sure the work out of the box for most people.
>
> gibi id building up
> https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co…
>
> as part of nova move to encode this but our goal is that
> deployment
> tools shoudl not need to be modifyed to tune these
> valued by defualt.
>
> >
> > So it will be kinda documented in a way after all.
> >
> >
> > told for a given release deploy this way.
> >
> > this is an internal impelation detail however we are not
> prepared to
> > deprecate usign eventlet until we are convicned
> >
> > that we can run properly without it.
> >
> > > For beginners, this would be a horrible nightmare if
> default
> > options
> > > simply wouldn't work. We *must* ship OpenStack working
> by default.
> > no one is suggesting we do otherwise.
> > >
> > > Cheers,
> > >
> > > Thomas Goirand (zigo)
> > >
> >
>
6 months, 2 weeks