On Wed, 2024-08-14 at 23:06 +0100, smooney@redhat.com wrote:
> On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote:
> > Thanks to both of you for the response . I try to insert pdb breakpoint at
> > some point in the code files and start the service. Then I issue a nova
> > command to check how the flow works. From then on I use the pdb commands to
> > check the function calls. This to me gives a very convenient way of
> > learning the code connections, since I don't have an idea of how these
> > functions and modules r related.  U people as developers have a design to
> > start with and work on from there. So u may know the purpose of each
> > function, class, etc.
> >
> > Since nova api doesn't support pdb becos of eventlet design, I learned to
> > print those objects and find their information.
> >
> > But with pdb, it's a smooth flow to inspect the class attributes, their
> > functions, arguments, types etc. I can even test those objects the python
> > way since pdb supports python prompt as well. Of course I do all this on
> > dev systems though...
>
> this is going to be a long responce and its a bit rambely but im goign to provide
> you with the context and poitn er try and figure out how to do this for your self.
>
> first thing to note is while its possibel it not trivial and the payoff is proably not
> worth it in the long run. we do not have up to day docs for how to do this because
> this is not how peole learn, contibute to ro develop nova in genral.
>
>
> nova does have a remote debug facilitate that is sepreate for the eventlet backdor to enable
> you to use an ide to debug remotely.
> that was added around 2012 and has worked on an off to vering degrees since then.
>
> the first thing to understand about nova and several other project is that they are a distibuted system
> i.e. the applciaiton that compise a given service like nova run in multiple service which are interconnected
> vai a message bus with in a service and rest apis are exposed for inter service comunication.
>
> when you enable a debugger and stop on a breakpoitn only one of the process is stop and the other contiue
> to exsculte which means that RPC calls, heathbeats and other asynconos tasks can and will timeout and fail.
>
> you cannot debug a distibuted system in teh same way as you woudl a simple applction where all state
> is executed in a signle main loop.
>
> the act of observing the internal behavior of the nova api or nova-comptue agent will change its behavior.
>
> nova has suppoort for remote debugging that was added about 12 years ago
> https://wiki.openstack.org/wiki/Nova/RemoteDebugging
> https://github.com/openstack/nova/blob/master/nova/debugger.py
>
> with that supprot you can use an idea like pycharm to connect to the remote debugger and attempt to
> debug a process.
>
> im currently propsoing removing that as part of the removal of eventlet partly because it shoudl not be needed
> when we dont have enventlet and mainly because it didnt work very well when i last used it.
>
> some ides have supprot fo gevent
>
> gevent is a fork of eventlet which was forked form some of hte libiares in stackless python
> eventlet converts a single thread python interperater in a cooperative concurent multi userland process.
>
by the way i forgot to say if you want to debug eventlet mokey patch code
adn your using a debugger that supprot gevent it mostly work with eventlet too if you enabel the gevnet
supprot in pycham which is disabeld by defaut.

i have had it workk reallly well for months and then not work for years and suddenly start workign agian.
so the proablme with usign a debugger with nova is not getting it working initally its keepign it working
and setting it up on a new machine if you create a new devstack vm for dev again.

its not hard, its hard to do repatbly, so blogs, docs and even automation
tends to bit wroth quickly.

> what does that mean , each python function is not operating in a seperate os stackframe, it has heap allcoated
> stack frames that allow context swtichs between userland thread at any tiem a funnction woudl do blocking io
>
> the reason single step debugging does not work well with gdb is that when you single step the eventlet engine
> is often going to end up context switching into the midel of another function.
>
> to work around this you just set breakpoint at the line you want to go to next and use continue instead fo single step
>
> one of the rules when using eventlet is that if your going to monkey patch  you need to monkey patch everyting early
>
> in addtion too the remote debuging facilites we can disabl mokey patching entirly.
> you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87
>
> this will entirly break nova-compute and several other nova process as they have
> effectivly infinity loops to pool for rpc messags, monitor hyperviors ectra.
>
> if the remote debugger is used it disables patching the thread lib
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46
>
> that is prartly why usign the remote debugger is kind fo buggy in itsself.
> the code is not inteded to work that way.
>
> these fucntionlatiy mainly exist to allow debuging tests not the actulaly running code.
>
> if i need to debug test i normaly jsut comment out
> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89
> and run the test in my ides(pycharm,vscodes) test debbuger.
> i normally code in nano/emacs but is do use ides if i really cant figure out what happening
> however i genreally start with  just adding log lines.
>
> when i first started workign on opentack in 2013 i very much liked using an ide and debugger
> and i woudl often spend time trying to get nova to work in a debugger.
>
> over tiem i generally found that that was not worth the effort.
> keepign a devstack deployment working that way was often more work then doing print staement
> or better writing functional or unit test to repoduce the problem.
>
> you previous mails ask about debuging nova's api.
>
> if you want to do that its one of the easiest thigns to debug.
> first do _not_ use the wsgi script
> adding apache of modwsgi is just going to make your life harder.
>
> just disable eventlet monkey patching, by commetiing or usign the env varible
> and run the nova-api console script directly under pdb or your debugger of choice.
> that will allow you to basiclly single step debug without any issues as it will not be using eventlet
> anymore and it has not infinitly loops or perodic tasks so it mostly just works when not using eventlet.
>
> that becase it didnt use eventlet at all in the api until about 2016 and even today it only really uses it in one
> place
> in my eventlet removal serise i split nova in to the part that use eventlet directly and the parts that dont in the
> second path
>
> https://review.opendev.org/c/openstack/nova/+/904424
>
> anything under nova/cmd/eventlet/ needs eventlet today
> everything under nova/cmd/standalone/ does not.
>
> the approch of doign an api call and following it in a debugger is not a good way in my expericne to learn how
> somethign
> like nova really works. it can hellp to a limtied degree but you cant eaislly debug across multipel processes.
> a minium nova deployment has 4 process typically more.
>
> you can try but each openstack service (nova, neutorn, cinder, ...) is typically compised of several micocservices
> some like keystone and palcement are just rest apis in front of a db and dont use eventlet at all so are trivial to
> debug as a result. the rest have non tirival runtime interactions that cant be as eaislly decompsed.
> becuase of that nova and other project invest in our functional tests to allwo ues to emulate that multi process env
> and better reason about the code.
>
> the debugging functionality is documetned in the contibutor guide
> https://github.com/openstack/nova/blob/master/doc/source/contributor/development-environment.rst#using-a-remote-debugger
> but we dont really adverstise that because none of the core maintainers use that any more and it has not been
> maintained
> for the better part of a decade. since we cant use this type of debugging with our ci system
> if we cant debug why a failure happens form the ci logs we add more logs because thyat will be useful for future us
> and operators in production. so our focus is on logging/monitoring for production rather then for developer eases
> simply because of the large techdebt that eventlet creates in this aready.
>
> i know nuetorn has had some better look with using debuggers then nova has,
>
> keystone/placmnet are eventlest free adn are just python web apps/rest apis so should "just work"
> in any python debugger.
>
> nova is prehaps the hardest to debug because of the legacy of the proejct so its not the best palce to start looking
> how this works. too be clear i have ran nova api, nova-conductor, nova scheduler and nova-compute all in 4 ide windows
> with 4 debuggers at once
> it was proably 3 year of working on openstack before i understood how to hack that ot work and since i learned
> how to properly debug without a debuggger, logs, code inspection, unit/fucntioal testing
> i have nver really need to do that again.
>
>
> gaining the ablity to run openstack in a debugger simply again, in a maintainable way is deffienly one of the reason
> im
> looking forward to removing eventlet but its still a lot of work.
> im not sure how helpful this was but this is proably as clsoe to a blog post as you will get.
>
> >
> > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi@yuggoth.org> wrote:
> >
> > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote:
> > > > How many more releases are you planning to include eventlet in
> > > > them? Seems they are removing any support for further development
> > > > according to their official website. So, is the community thinking
> > > > of any alternatives?
> > > [...]
> > >
> > > Just to expand on the first part of Sean's reply, most of what you
> > > want to know is probably answered here:
> > >
> > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html
> > >
> > > --
> > > Jeremy Stanley
> > >
>