Search results for "#eventlet-removal" - openstack-discuss

Re: Eventlet and debugging

by smooney＠redhat.com

On Wed, 2024-08-14 at 23:06 +0100, smooney(a)redhat.com wrote: > On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote: > > Thanks to both of you for the response . I try to insert pdb breakpoint at > > some point in the code files and start the service. Then I issue a nova > > command to check how the flow works. From then on I use the pdb commands to > > check the function calls. This to me gives a very convenient way of > > learning the code connections, since I don't have an idea of how these > > functions and modules r related. U people as developers have a design to > > start with and work on from there. So u may know the purpose of each > > function, class, etc. > > > > Since nova api doesn't support pdb becos of eventlet design, I learned to > > print those objects and find their information. > > > > But with pdb, it's a smooth flow to inspect the class attributes, their > > functions, arguments, types etc. I can even test those objects the python > > way since pdb supports python prompt as well. Of course I do all this on > > dev systems though... > > this is going to be a long responce and its a bit rambely but im goign to provide > you with the context and poitn er try and figure out how to do this for your self. > > first thing to note is while its possibel it not trivial and the payoff is proably not > worth it in the long run. we do not have up to day docs for how to do this because > this is not how peole learn, contibute to ro develop nova in genral. > > > nova does have a remote debug facilitate that is sepreate for the eventlet backdor to enable > you to use an ide to debug remotely. > that was added around 2012 and has worked on an off to vering degrees since then. > > the first thing to understand about nova and several other project is that they are a distibuted system > i.e. the applciaiton that compise a given service like nova run in multiple service which are interconnected > vai a message bus with in a service and rest apis are exposed for inter service comunication. > > when you enable a debugger and stop on a breakpoitn only one of the process is stop and the other contiue > to exsculte which means that RPC calls, heathbeats and other asynconos tasks can and will timeout and fail. > > you cannot debug a distibuted system in teh same way as you woudl a simple applction where all state > is executed in a signle main loop. > > the act of observing the internal behavior of the nova api or nova-comptue agent will change its behavior. > > nova has suppoort for remote debugging that was added about 12 years ago > https://wiki.openstack.org/wiki/Nova/RemoteDebugging > https://github.com/openstack/nova/blob/master/nova/debugger.py > > with that supprot you can use an idea like pycharm to connect to the remote debugger and attempt to > debug a process. > > im currently propsoing removing that as part of the removal of eventlet partly because it shoudl not be needed > when we dont have enventlet and mainly because it didnt work very well when i last used it. > > some ides have supprot fo gevent > > gevent is a fork of eventlet which was forked form some of hte libiares in stackless python > eventlet converts a single thread python interperater in a cooperative concurent multi userland process. > by the way i forgot to say if you want to debug eventlet mokey patch code adn your using a debugger that supprot gevent it mostly work with eventlet too if you enabel the gevnet supprot in pycham which is disabeld by defaut. i have had it workk reallly well for months and then not work for years and suddenly start workign agian. so the proablme with usign a debugger with nova is not getting it working initally its keepign it working and setting it up on a new machine if you create a new devstack vm for dev again. its not hard, its hard to do repatbly, so blogs, docs and even automation tends to bit wroth quickly. > what does that mean , each python function is not operating in a seperate os stackframe, it has heap allcoated > stack frames that allow context swtichs between userland thread at any tiem a funnction woudl do blocking io > > the reason single step debugging does not work well with gdb is that when you single step the eventlet engine > is often going to end up context switching into the midel of another function. > > to work around this you just set breakpoint at the line you want to go to next and use continue instead fo single step > > one of the rules when using eventlet is that if your going to monkey patch you need to monkey patch everyting early > > in addtion too the remote debuging facilites we can disabl mokey patching entirly. > you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1 > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87 > > this will entirly break nova-compute and several other nova process as they have > effectivly infinity loops to pool for rpc messags, monitor hyperviors ectra. > > if the remote debugger is used it disables patching the thread lib > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46 > > that is prartly why usign the remote debugger is kind fo buggy in itsself. > the code is not inteded to work that way. > > these fucntionlatiy mainly exist to allow debuging tests not the actulaly running code. > > if i need to debug test i normaly jsut comment out > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89 > and run the test in my ides(pycharm,vscodes) test debbuger. > i normally code in nano/emacs but is do use ides if i really cant figure out what happening > however i genreally start with just adding log lines. > > when i first started workign on opentack in 2013 i very much liked using an ide and debugger > and i woudl often spend time trying to get nova to work in a debugger. > > over tiem i generally found that that was not worth the effort. > keepign a devstack deployment working that way was often more work then doing print staement > or better writing functional or unit test to repoduce the problem. > > you previous mails ask about debuging nova's api. > > if you want to do that its one of the easiest thigns to debug. > first do _not_ use the wsgi script > adding apache of modwsgi is just going to make your life harder. > > just disable eventlet monkey patching, by commetiing or usign the env varible > and run the nova-api console script directly under pdb or your debugger of choice. > that will allow you to basiclly single step debug without any issues as it will not be using eventlet > anymore and it has not infinitly loops or perodic tasks so it mostly just works when not using eventlet. > > that becase it didnt use eventlet at all in the api until about 2016 and even today it only really uses it in one > place > in my eventlet removal serise i split nova in to the part that use eventlet directly and the parts that dont in the > second path > > https://review.opendev.org/c/openstack/nova/+/904424 > > anything under nova/cmd/eventlet/ needs eventlet today > everything under nova/cmd/standalone/ does not. > > the approch of doign an api call and following it in a debugger is not a good way in my expericne to learn how > somethign > like nova really works. it can hellp to a limtied degree but you cant eaislly debug across multipel processes. > a minium nova deployment has 4 process typically more. > > you can try but each openstack service (nova, neutorn, cinder, ...) is typically compised of several micocservices > some like keystone and palcement are just rest apis in front of a db and dont use eventlet at all so are trivial to > debug as a result. the rest have non tirival runtime interactions that cant be as eaislly decompsed. > becuase of that nova and other project invest in our functional tests to allwo ues to emulate that multi process env > and better reason about the code. > > the debugging functionality is documetned in the contibutor guide > https://github.com/openstack/nova/blob/master/doc/source/contributor/develo… > but we dont really adverstise that because none of the core maintainers use that any more and it has not been > maintained > for the better part of a decade. since we cant use this type of debugging with our ci system > if we cant debug why a failure happens form the ci logs we add more logs because thyat will be useful for future us > and operators in production. so our focus is on logging/monitoring for production rather then for developer eases > simply because of the large techdebt that eventlet creates in this aready. > > i know nuetorn has had some better look with using debuggers then nova has, > > keystone/placmnet are eventlest free adn are just python web apps/rest apis so should "just work" > in any python debugger. > > nova is prehaps the hardest to debug because of the legacy of the proejct so its not the best palce to start looking > how this works. too be clear i have ran nova api, nova-conductor, nova scheduler and nova-compute all in 4 ide windows > with 4 debuggers at once > it was proably 3 year of working on openstack before i understood how to hack that ot work and since i learned > how to properly debug without a debuggger, logs, code inspection, unit/fucntioal testing > i have nver really need to do that again. > > > gaining the ablity to run openstack in a debugger simply again, in a maintainable way is deffienly one of the reason > im > looking forward to removing eventlet but its still a lot of work. > im not sure how helpful this was but this is proably as clsoe to a blog post as you will get. > > > > > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote: > > > > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote: > > > > How many more releases are you planning to include eventlet in > > > > them? Seems they are removing any support for further development > > > > according to their official website. So, is the community thinking > > > > of any alternatives? > > > [...] > > > > > > Just to expand on the first part of Sean's reply, most of what you > > > want to know is probably answered here: > > > > > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html > > > > > > -- > > > Jeremy Stanley > > > >

11 months, 4 weeks

Re: Eventlet and debugging

by Mike Carden

In almost a decade of reading smooney(a)redhat.com's Lysdexic posts, this is the longest one I have ever read. No criticism intended at all - none whatsoever. BUT... I am still absolutely Gobsmacked that anyone can possibly write even vaguely working code, if their written English communications are so comprehensively broken. How does anyone get away with ignoring Spelling and Grammar in Code? I love your work Sean, but I do struggle to understand almost everything you type. -- MC On Thu, 15 Aug 2024 at 08:07, <smooney(a)redhat.com> wrote: > On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote: > > Thanks to both of you for the response . I try to insert pdb breakpoint > at > > some point in the code files and start the service. Then I issue a nova > > command to check how the flow works. From then on I use the pdb commands > to > > check the function calls. This to me gives a very convenient way of > > learning the code connections, since I don't have an idea of how these > > functions and modules r related. U people as developers have a design to > > start with and work on from there. So u may know the purpose of each > > function, class, etc. > > > > Since nova api doesn't support pdb becos of eventlet design, I learned to > > print those objects and find their information. > > > > But with pdb, it's a smooth flow to inspect the class attributes, their > > functions, arguments, types etc. I can even test those objects the python > > way since pdb supports python prompt as well. Of course I do all this on > > dev systems though... > > this is going to be a long responce and its a bit rambely but im goign to > provide > you with the context and poitn er try and figure out how to do this for > your self. > > first thing to note is while its possibel it not trivial and the payoff is > proably not > worth it in the long run. we do not have up to day docs for how to do this > because > this is not how peole learn, contibute to ro develop nova in genral. > > > nova does have a remote debug facilitate that is sepreate for the eventlet > backdor to enable > you to use an ide to debug remotely. > that was added around 2012 and has worked on an off to vering degrees > since then. > > the first thing to understand about nova and several other project is that > they are a distibuted system > i.e. the applciaiton that compise a given service like nova run in > multiple service which are interconnected > vai a message bus with in a service and rest apis are exposed for inter > service comunication. > > when you enable a debugger and stop on a breakpoitn only one of the > process is stop and the other contiue > to exsculte which means that RPC calls, heathbeats and other asynconos > tasks can and will timeout and fail. > > you cannot debug a distibuted system in teh same way as you woudl a simple > applction where all state > is executed in a signle main loop. > > the act of observing the internal behavior of the nova api or nova-comptue > agent will change its behavior. > > nova has suppoort for remote debugging that was added about 12 years ago > https://wiki.openstack.org/wiki/Nova/RemoteDebugging > https://github.com/openstack/nova/blob/master/nova/debugger.py > > with that supprot you can use an idea like pycharm to connect to the > remote debugger and attempt to > debug a process. > > im currently propsoing removing that as part of the removal of eventlet > partly because it shoudl not be needed > when we dont have enventlet and mainly because it didnt work very well > when i last used it. > > some ides have supprot fo gevent > > gevent is a fork of eventlet which was forked form some of hte libiares in > stackless python > eventlet converts a single thread python interperater in a cooperative > concurent multi userland process. > > what does that mean , each python function is not operating in a seperate > os stackframe, it has heap allcoated > stack frames that allow context swtichs between userland thread at any > tiem a funnction woudl do blocking io > > the reason single step debugging does not work well with gdb is that when > you single step the eventlet engine > is often going to end up context switching into the midel of another > function. > > to work around this you just set breakpoint at the line you want to go to > next and use continue instead fo single step > > one of the rules when using eventlet is that if your going to monkey > patch you need to monkey patch everyting early > > in addtion too the remote debuging facilites we can disabl mokey patching > entirly. > you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1 > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87 > > this will entirly break nova-compute and several other nova process as > they have > effectivly infinity loops to pool for rpc messags, monitor hyperviors > ectra. > > if the remote debugger is used it disables patching the thread lib > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46 > > that is prartly why usign the remote debugger is kind fo buggy in itsself. > the code is not inteded to work that way. > > these fucntionlatiy mainly exist to allow debuging tests not the actulaly > running code. > > if i need to debug test i normaly jsut comment out > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89 > and run the test in my ides(pycharm,vscodes) test debbuger. > i normally code in nano/emacs but is do use ides if i really cant figure > out what happening > however i genreally start with just adding log lines. > > when i first started workign on opentack in 2013 i very much liked using > an ide and debugger > and i woudl often spend time trying to get nova to work in a debugger. > > over tiem i generally found that that was not worth the effort. > keepign a devstack deployment working that way was often more work then > doing print staement > or better writing functional or unit test to repoduce the problem. > > you previous mails ask about debuging nova's api. > > if you want to do that its one of the easiest thigns to debug. > first do _not_ use the wsgi script > adding apache of modwsgi is just going to make your life harder. > > just disable eventlet monkey patching, by commetiing or usign the env > varible > and run the nova-api console script directly under pdb or your debugger of > choice. > that will allow you to basiclly single step debug without any issues as it > will not be using eventlet > anymore and it has not infinitly loops or perodic tasks so it mostly just > works when not using eventlet. > > that becase it didnt use eventlet at all in the api until about 2016 and > even today it only really uses it in one place > in my eventlet removal serise i split nova in to the part that use > eventlet directly and the parts that dont in the > second path > > https://review.opendev.org/c/openstack/nova/+/904424 > > anything under nova/cmd/eventlet/ needs eventlet today > everything under nova/cmd/standalone/ does not. > > the approch of doign an api call and following it in a debugger is not a > good way in my expericne to learn how somethign > like nova really works. it can hellp to a limtied degree but you cant > eaislly debug across multipel processes. > a minium nova deployment has 4 process typically more. > > you can try but each openstack service (nova, neutorn, cinder, ...) is > typically compised of several micocservices > some like keystone and palcement are just rest apis in front of a db and > dont use eventlet at all so are trivial to > debug as a result. the rest have non tirival runtime interactions that > cant be as eaislly decompsed. > becuase of that nova and other project invest in our functional tests to > allwo ues to emulate that multi process env > and better reason about the code. > > the debugging functionality is documetned in the contibutor guide > > https://github.com/openstack/nova/blob/master/doc/source/contributor/develo… > but we dont really adverstise that because none of the core maintainers > use that any more and it has not been maintained > for the better part of a decade. since we cant use this type of debugging > with our ci system > if we cant debug why a failure happens form the ci logs we add more logs > because thyat will be useful for future us > and operators in production. so our focus is on logging/monitoring for > production rather then for developer eases > simply because of the large techdebt that eventlet creates in this aready. > > i know nuetorn has had some better look with using debuggers then nova has, > > keystone/placmnet are eventlest free adn are just python web apps/rest > apis so should "just work" > in any python debugger. > > nova is prehaps the hardest to debug because of the legacy of the proejct > so its not the best palce to start looking > how this works. too be clear i have ran nova api, nova-conductor, nova > scheduler and nova-compute all in 4 ide windows > with 4 debuggers at once > it was proably 3 year of working on openstack before i understood how to > hack that ot work and since i learned > how to properly debug without a debuggger, logs, code inspection, > unit/fucntioal testing > i have nver really need to do that again. > > > gaining the ablity to run openstack in a debugger simply again, in a > maintainable way is deffienly one of the reason im > looking forward to removing eventlet but its still a lot of work. > im not sure how helpful this was but this is proably as clsoe to a blog > post as you will get. > > > > > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote: > > > > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote: > > > > How many more releases are you planning to include eventlet in > > > > them? Seems they are removing any support for further development > > > > according to their official website. So, is the community thinking > > > > of any alternatives? > > > [...] > > > > > > Just to expand on the first part of Sean's reply, most of what you > > > want to know is probably answered here: > > > > > > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html > > > > > > -- > > > Jeremy Stanley > > > > >

11 months, 4 weeks

Re: [eventlet-removal]When to drop eventlet support

by Dmitriy Rabotyagov

In case you try to use a 32gb box with 16 cores as a controller for OpenStack - it will blow off with default amount of workers for wsgi and /or eventlet apps. While you can argue this should not be used as production setup, this can be totally valid for sandboxes and we wanna provide consistent and reliable behavior for users. But my argument was not in if/how we want to fine-tune deployments, but also understand and provide means to define what's needed as well as potential ability to revert in worst case scenario as a temporary workaround. So still some variables and logic would be introduced from what I understand today. On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote: > > On 16/06/2025 13:27, Dmitriy Rabotyagov wrote: > > > > > > > > sayint its FUD is not helpful. > > > > we got a driect ask form operator and soem core to not do a hard > > switch > > over. > > > > and while i wanted to only support one model for each binary at a > > time > > we were sepcificly ask to make it configurable. > > > > > In the later case, your only available action is help fixing > > bugs. It > > > is not up to the operators to double-guess what may or may not > > work. > > > > correct we are not planning to document how to change mode we were > > planning to only use this configuration in ci and operator would be > > > > > > Well, we'd need to have that communicated so that deployment toolings > > could adopt their setup to changes, as, for instance, in OSA amount of > > eventlet workers are calculated based on the system facts, so we'd > > need to change the logic and also suggest how users should treat this > > new logic for their systems. > > why is OSA doing that at all today? > we generally don't recommend changing those values from the default > unless you really know what your doing. > i don't think other installer do that. > tripleo, kolla-ansbile and our new golang based installer do not, nor > does devstack so its surprising to me that OSA would change such low > level values > by default. > > we will document any new config options we and and we are documentation > how to tune the new options for thread pools but we do not expect > installation > tools to modify them by default. we are explicitly not making the > options based on the amount of resources on the host i.e. dynamically > calculated based > on the number of CPU cores. > > for example we are explicitly setting the number of scatter_gather > thread in the the dedicated thread pool to 5 > why its a nice small number that will work for most people out of the box. > > can you adjust it, yes but it scale based on the number of nova cells > you have an 99% wont have more then 5 cells. > > using information about the host where the API is deployed to infer the > value of that would be incorrect. > > you can really only make an informed decision about how to tune that > based on monitoring the usage of the pool. > > that how we expect most of the other tuning options to go as well. > > our defaults in nova tend to be higher then you would actually need in a > real environment so while it may make sense to reduce > them we try to make sure the work out of the box for most people. > > gibi id building up > > https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co… > > as part of nova move to encode this but our goal is that deployment > tools shoudl not need to be modifyed to tune these > valued by defualt. > > > > > So it will be kinda documented in a way after all. > > > > > > told for a given release deploy this way. > > > > this is an internal impelation detail however we are not prepared to > > deprecate usign eventlet until we are convicned > > > > that we can run properly without it. > > > > > For beginners, this would be a horrible nightmare if default > > options > > > simply wouldn't work. We *must* ship OpenStack working by default. > > no one is suggesting we do otherwise. > > > > > > Cheers, > > > > > > Thomas Goirand (zigo) > > > > > > >

1 month, 3 weeks

Re: Eventlet and debugging

by engineer2024

Appreciate taking time to post this. I will look into the options and try them... Have a nice week ahead On Thu, 15 Aug 2024, 03:53 , <smooney(a)redhat.com> wrote: > On Wed, 2024-08-14 at 23:06 +0100, smooney(a)redhat.com wrote: > > On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote: > > > Thanks to both of you for the response . I try to insert pdb > breakpoint at > > > some point in the code files and start the service. Then I issue a nova > > > command to check how the flow works. From then on I use the pdb > commands to > > > check the function calls. This to me gives a very convenient way of > > > learning the code connections, since I don't have an idea of how these > > > functions and modules r related. U people as developers have a design > to > > > start with and work on from there. So u may know the purpose of each > > > function, class, etc. > > > > > > Since nova api doesn't support pdb becos of eventlet design, I learned > to > > > print those objects and find their information. > > > > > > But with pdb, it's a smooth flow to inspect the class attributes, their > > > functions, arguments, types etc. I can even test those objects the > python > > > way since pdb supports python prompt as well. Of course I do all this > on > > > dev systems though... > > > > this is going to be a long responce and its a bit rambely but im goign > to provide > > you with the context and poitn er try and figure out how to do this for > your self. > > > > first thing to note is while its possibel it not trivial and the payoff > is proably not > > worth it in the long run. we do not have up to day docs for how to do > this because > > this is not how peole learn, contibute to ro develop nova in genral. > > > > > > nova does have a remote debug facilitate that is sepreate for the > eventlet backdor to enable > > you to use an ide to debug remotely. > > that was added around 2012 and has worked on an off to vering degrees > since then. > > > > the first thing to understand about nova and several other project is > that they are a distibuted system > > i.e. the applciaiton that compise a given service like nova run in > multiple service which are interconnected > > vai a message bus with in a service and rest apis are exposed for inter > service comunication. > > > > when you enable a debugger and stop on a breakpoitn only one of the > process is stop and the other contiue > > to exsculte which means that RPC calls, heathbeats and other asynconos > tasks can and will timeout and fail. > > > > you cannot debug a distibuted system in teh same way as you woudl a > simple applction where all state > > is executed in a signle main loop. > > > > the act of observing the internal behavior of the nova api or > nova-comptue agent will change its behavior. > > > > nova has suppoort for remote debugging that was added about 12 years ago > > https://wiki.openstack.org/wiki/Nova/RemoteDebugging > > https://github.com/openstack/nova/blob/master/nova/debugger.py > > > > with that supprot you can use an idea like pycharm to connect to the > remote debugger and attempt to > > debug a process. > > > > im currently propsoing removing that as part of the removal of eventlet > partly because it shoudl not be needed > > when we dont have enventlet and mainly because it didnt work very well > when i last used it. > > > > some ides have supprot fo gevent > > > > gevent is a fork of eventlet which was forked form some of hte libiares > in stackless python > > eventlet converts a single thread python interperater in a cooperative > concurent multi userland process. > > > by the way i forgot to say if you want to debug eventlet mokey patch code > adn your using a debugger that supprot gevent it mostly work with eventlet > too if you enabel the gevnet > supprot in pycham which is disabeld by defaut. > > i have had it workk reallly well for months and then not work for years > and suddenly start workign agian. > so the proablme with usign a debugger with nova is not getting it working > initally its keepign it working > and setting it up on a new machine if you create a new devstack vm for dev > again. > > its not hard, its hard to do repatbly, so blogs, docs and even automation > tends to bit wroth quickly. > > > what does that mean , each python function is not operating in a > seperate os stackframe, it has heap allcoated > > stack frames that allow context swtichs between userland thread at any > tiem a funnction woudl do blocking io > > > > the reason single step debugging does not work well with gdb is that > when you single step the eventlet engine > > is often going to end up context switching into the midel of another > function. > > > > to work around this you just set breakpoint at the line you want to go > to next and use continue instead fo single step > > > > one of the rules when using eventlet is that if your going to monkey > patch you need to monkey patch everyting early > > > > in addtion too the remote debuging facilites we can disabl mokey > patching entirly. > > you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1 > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87 > > > > this will entirly break nova-compute and several other nova process as > they have > > effectivly infinity loops to pool for rpc messags, monitor hyperviors > ectra. > > > > if the remote debugger is used it disables patching the thread lib > > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46 > > > > that is prartly why usign the remote debugger is kind fo buggy in > itsself. > > the code is not inteded to work that way. > > > > these fucntionlatiy mainly exist to allow debuging tests not the > actulaly running code. > > > > if i need to debug test i normaly jsut comment out > > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89 > > and run the test in my ides(pycharm,vscodes) test debbuger. > > i normally code in nano/emacs but is do use ides if i really cant figure > out what happening > > however i genreally start with just adding log lines. > > > > when i first started workign on opentack in 2013 i very much liked using > an ide and debugger > > and i woudl often spend time trying to get nova to work in a debugger. > > > > over tiem i generally found that that was not worth the effort. > > keepign a devstack deployment working that way was often more work then > doing print staement > > or better writing functional or unit test to repoduce the problem. > > > > you previous mails ask about debuging nova's api. > > > > if you want to do that its one of the easiest thigns to debug. > > first do _not_ use the wsgi script > > adding apache of modwsgi is just going to make your life harder. > > > > just disable eventlet monkey patching, by commetiing or usign the env > varible > > and run the nova-api console script directly under pdb or your debugger > of choice. > > that will allow you to basiclly single step debug without any issues as > it will not be using eventlet > > anymore and it has not infinitly loops or perodic tasks so it mostly > just works when not using eventlet. > > > > that becase it didnt use eventlet at all in the api until about 2016 and > even today it only really uses it in one > > place > > in my eventlet removal serise i split nova in to the part that use > eventlet directly and the parts that dont in the > > second path > > > > https://review.opendev.org/c/openstack/nova/+/904424 > > > > anything under nova/cmd/eventlet/ needs eventlet today > > everything under nova/cmd/standalone/ does not. > > > > the approch of doign an api call and following it in a debugger is not a > good way in my expericne to learn how > > somethign > > like nova really works. it can hellp to a limtied degree but you cant > eaislly debug across multipel processes. > > a minium nova deployment has 4 process typically more. > > > > you can try but each openstack service (nova, neutorn, cinder, ...) is > typically compised of several micocservices > > some like keystone and palcement are just rest apis in front of a db and > dont use eventlet at all so are trivial to > > debug as a result. the rest have non tirival runtime interactions that > cant be as eaislly decompsed. > > becuase of that nova and other project invest in our functional tests to > allwo ues to emulate that multi process env > > and better reason about the code. > > > > the debugging functionality is documetned in the contibutor guide > > > https://github.com/openstack/nova/blob/master/doc/source/contributor/develo… > > but we dont really adverstise that because none of the core maintainers > use that any more and it has not been > > maintained > > for the better part of a decade. since we cant use this type of > debugging with our ci system > > if we cant debug why a failure happens form the ci logs we add more logs > because thyat will be useful for future us > > and operators in production. so our focus is on logging/monitoring for > production rather then for developer eases > > simply because of the large techdebt that eventlet creates in this > aready. > > > > i know nuetorn has had some better look with using debuggers then nova > has, > > > > keystone/placmnet are eventlest free adn are just python web apps/rest > apis so should "just work" > > in any python debugger. > > > > nova is prehaps the hardest to debug because of the legacy of the > proejct so its not the best palce to start looking > > how this works. too be clear i have ran nova api, nova-conductor, nova > scheduler and nova-compute all in 4 ide windows > > with 4 debuggers at once > > it was proably 3 year of working on openstack before i understood how to > hack that ot work and since i learned > > how to properly debug without a debuggger, logs, code inspection, > unit/fucntioal testing > > i have nver really need to do that again. > > > > > > gaining the ablity to run openstack in a debugger simply again, in a > maintainable way is deffienly one of the reason > > im > > looking forward to removing eventlet but its still a lot of work. > > im not sure how helpful this was but this is proably as clsoe to a blog > post as you will get. > > > > > > > > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote: > > > > > > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote: > > > > > How many more releases are you planning to include eventlet in > > > > > them? Seems they are removing any support for further development > > > > > according to their official website. So, is the community thinking > > > > > of any alternatives? > > > > [...] > > > > > > > > Just to expand on the first part of Sean's reply, most of what you > > > > want to know is probably answered here: > > > > > > > > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html > > > > > > > > -- > > > > Jeremy Stanley > > > > > > > >

11 months, 4 weeks

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-9)

by Goutham Pacha Ravi

Hello Stackers, We're counting down 9 weeks to the 2024.2 (Dalmatian) release [1]. OpenStack projects that follow the cycle-with-rc release model [2] will have their code freeze in 4 weeks. Between now and the release, the release team will communicate with us weekly to coordinate deliverables across the community. The elections for OpenStack PTLs and Technical Committee members are around the corner as well, so now is a good time to ensure that you are eligible to vote in these elections! Please navigate to: https://civs1.civs.us/cgi-bin/opt_in.pl Enter your preferred email address on Gerrit and request activation. It only takes a minute! :) The following are updates from the OpenStack Technical Committee during the past week. === Weekly Meeting === The TC had a productive meeting via IRC on OFTC's #openstack-tc channel last week [3]. We discussed follow-ups for the eventlet removal goal proposal [4] and recent updates to inactive and emerging OpenStack projects. We spent some time assessing gate health concerns, including an issue with a change to `setuptools` that requires us to bump the upper constraint on the "packaging" library in stable/2023.1 [5]. This was important to get some CVE fixes landed in various OpenStack code repositories. Finally, we refreshed our understanding of the TC's house rules. To summarize: - Changes to the TC charter require at least 2/3 of the TC voting in favor, while governance motions require at least 1/3 of the TC to vote in favor. - TC members can vote positively, negatively, or abstain (using the “RollCall-Vote” in Gerrit). - Decisions need more positive votes than negative votes (ties mean the proposal is rejected). - Any motion or charter-change is deliberately open at least a week after it was last proposed/updated, allowing the TC and the wider community to participate fully. - Significant governance changes take a while to merge because of the importance of ensuring full TC and community engagement in votes. Our next weekly meeting is tomorrow (2024-07-30) at 1800 UTC via IRC on OFTC's #openstack-tc channel. You can find the agenda on the OpenStack TC wiki page [6]. Please consider joining us and remember, you can edit the wiki and propose new topics! === Governance Proposals === ==== Merged ==== - Add `os-test-images` under glance | https://review.opendev.org/c/openstack/governance/+/925044 ==== Open for review/discussion ==== - [Follow-up] Fix typos in the eventlet proposal | https://review.opendev.org/c/openstack/governance/+/924944 - Update criteria for inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 - Follow up on the affiliation diversity handling | https://review.opendev.org/c/openstack/governance/+/923876 - Inactive state extensions: Monasca | https://review.opendev.org/c/openstack/governance/+/923919 === Upcoming Events === - 2024-08-12: Call for Proposals due for the OpenInfra Days NA [7] - 2024-08-14: Nominations open for 2025.1 OpenStack PTL+TC elections - 2024-08-24: OpenInfra Days, Vietnam: https://2024.vietopeninfra.org/ - 2024-08-29: Dalmatian-3 milestone - 2024-09-03: OpenInfra Summit Asia, Suwon The OpenStack Technical Committee serves you! We're always willing to chat and would love feedback: === How to contact the TC === You can reach out to the TC in multiple ways: 1. Email: Send an email with the tag [tc] on this email list. 2. Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel. 3. Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [7]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] OpenStack cycle-with-rc model: https://releases.openstack.org/reference/release_models.html#cycle-with-rc [3] TC Meeting IRC Logs, 2024-07-23: https://meetings.opendev.org/meetings/tc/2024/tc.2024-07-23-18.00.log.html [4] Remove Eventlet From Openstack: https://review.opendev.org/c/openstack/governance/+/902585 [5] Update packaging constraint in stable/2023.1: https://review.opendev.org/c/openstack/requirements/+/924764 [6] TC Meeting, 2024-07-30: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [7] Call for Proposals, OpenInfra Days NA: https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc…

1 year

Re: Eventlet and debugging

by engineer2024

Actually, when I was in my graduate/post graduate studies in computer science, I used to think that whoever is technically smart but unable to communicate in english would find a hard time in the industry. But after starting my career in IT and seeing many brilliant minds, I am convinced that language is not as important as the skill itself. We just need to understand each other in anyway possible, not only in english language. So many of my colleagues have gone on to higher positions in IT now despite their lack of proper english communication. So I think if you are able to express your ideas in any form, thats enough. Yes its important for certain roles like sales , client/customer facing. But not so for engineers. And regarding Sean's typos, I can understand he is quick typing on a mobile device or so, that these catch up. No issues though... On Thu, Aug 15, 2024 at 1:06 PM Mike Carden <mike.carden(a)gmail.com> wrote: > In almost a decade of reading smooney(a)redhat.com's Lysdexic posts, this > is the longest one I have ever read. No criticism intended at all - none > whatsoever. > > BUT... I am still absolutely Gobsmacked that anyone can possibly write > even vaguely working code, if their written English communications are so > comprehensively broken. How does anyone get away with ignoring Spelling and > Grammar in Code? > > I love your work Sean, but I do struggle to understand almost everything > you type. > > -- > MC > > > > On Thu, 15 Aug 2024 at 08:07, <smooney(a)redhat.com> wrote: > >> On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote: >> > Thanks to both of you for the response . I try to insert pdb breakpoint >> at >> > some point in the code files and start the service. Then I issue a nova >> > command to check how the flow works. From then on I use the pdb >> commands to >> > check the function calls. This to me gives a very convenient way of >> > learning the code connections, since I don't have an idea of how these >> > functions and modules r related. U people as developers have a design >> to >> > start with and work on from there. So u may know the purpose of each >> > function, class, etc. >> > >> > Since nova api doesn't support pdb becos of eventlet design, I learned >> to >> > print those objects and find their information. >> > >> > But with pdb, it's a smooth flow to inspect the class attributes, their >> > functions, arguments, types etc. I can even test those objects the >> python >> > way since pdb supports python prompt as well. Of course I do all this on >> > dev systems though... >> >> this is going to be a long responce and its a bit rambely but im goign to >> provide >> you with the context and poitn er try and figure out how to do this for >> your self. >> >> first thing to note is while its possibel it not trivial and the payoff >> is proably not >> worth it in the long run. we do not have up to day docs for how to do >> this because >> this is not how peole learn, contibute to ro develop nova in genral. >> >> >> nova does have a remote debug facilitate that is sepreate for the >> eventlet backdor to enable >> you to use an ide to debug remotely. >> that was added around 2012 and has worked on an off to vering degrees >> since then. >> >> the first thing to understand about nova and several other project is >> that they are a distibuted system >> i.e. the applciaiton that compise a given service like nova run in >> multiple service which are interconnected >> vai a message bus with in a service and rest apis are exposed for inter >> service comunication. >> >> when you enable a debugger and stop on a breakpoitn only one of the >> process is stop and the other contiue >> to exsculte which means that RPC calls, heathbeats and other asynconos >> tasks can and will timeout and fail. >> >> you cannot debug a distibuted system in teh same way as you woudl a >> simple applction where all state >> is executed in a signle main loop. >> >> the act of observing the internal behavior of the nova api or >> nova-comptue agent will change its behavior. >> >> nova has suppoort for remote debugging that was added about 12 years ago >> https://wiki.openstack.org/wiki/Nova/RemoteDebugging >> https://github.com/openstack/nova/blob/master/nova/debugger.py >> >> with that supprot you can use an idea like pycharm to connect to the >> remote debugger and attempt to >> debug a process. >> >> im currently propsoing removing that as part of the removal of eventlet >> partly because it shoudl not be needed >> when we dont have enventlet and mainly because it didnt work very well >> when i last used it. >> >> some ides have supprot fo gevent >> >> gevent is a fork of eventlet which was forked form some of hte libiares >> in stackless python >> eventlet converts a single thread python interperater in a cooperative >> concurent multi userland process. >> >> what does that mean , each python function is not operating in a seperate >> os stackframe, it has heap allcoated >> stack frames that allow context swtichs between userland thread at any >> tiem a funnction woudl do blocking io >> >> the reason single step debugging does not work well with gdb is that when >> you single step the eventlet engine >> is often going to end up context switching into the midel of another >> function. >> >> to work around this you just set breakpoint at the line you want to go to >> next and use continue instead fo single step >> >> one of the rules when using eventlet is that if your going to monkey >> patch you need to monkey patch everyting early >> >> in addtion too the remote debuging facilites we can disabl mokey patching >> entirly. >> you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1 >> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87 >> >> this will entirly break nova-compute and several other nova process as >> they have >> effectivly infinity loops to pool for rpc messags, monitor hyperviors >> ectra. >> >> if the remote debugger is used it disables patching the thread lib >> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46 >> >> that is prartly why usign the remote debugger is kind fo buggy in itsself. >> the code is not inteded to work that way. >> >> these fucntionlatiy mainly exist to allow debuging tests not the actulaly >> running code. >> >> if i need to debug test i normaly jsut comment out >> https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89 >> and run the test in my ides(pycharm,vscodes) test debbuger. >> i normally code in nano/emacs but is do use ides if i really cant figure >> out what happening >> however i genreally start with just adding log lines. >> >> when i first started workign on opentack in 2013 i very much liked using >> an ide and debugger >> and i woudl often spend time trying to get nova to work in a debugger. >> >> over tiem i generally found that that was not worth the effort. >> keepign a devstack deployment working that way was often more work then >> doing print staement >> or better writing functional or unit test to repoduce the problem. >> >> you previous mails ask about debuging nova's api. >> >> if you want to do that its one of the easiest thigns to debug. >> first do _not_ use the wsgi script >> adding apache of modwsgi is just going to make your life harder. >> >> just disable eventlet monkey patching, by commetiing or usign the env >> varible >> and run the nova-api console script directly under pdb or your debugger >> of choice. >> that will allow you to basiclly single step debug without any issues as >> it will not be using eventlet >> anymore and it has not infinitly loops or perodic tasks so it mostly just >> works when not using eventlet. >> >> that becase it didnt use eventlet at all in the api until about 2016 and >> even today it only really uses it in one place >> in my eventlet removal serise i split nova in to the part that use >> eventlet directly and the parts that dont in the >> second path >> >> https://review.opendev.org/c/openstack/nova/+/904424 >> >> anything under nova/cmd/eventlet/ needs eventlet today >> everything under nova/cmd/standalone/ does not. >> >> the approch of doign an api call and following it in a debugger is not a >> good way in my expericne to learn how somethign >> like nova really works. it can hellp to a limtied degree but you cant >> eaislly debug across multipel processes. >> a minium nova deployment has 4 process typically more. >> >> you can try but each openstack service (nova, neutorn, cinder, ...) is >> typically compised of several micocservices >> some like keystone and palcement are just rest apis in front of a db and >> dont use eventlet at all so are trivial to >> debug as a result. the rest have non tirival runtime interactions that >> cant be as eaislly decompsed. >> becuase of that nova and other project invest in our functional tests to >> allwo ues to emulate that multi process env >> and better reason about the code. >> >> the debugging functionality is documetned in the contibutor guide >> >> https://github.com/openstack/nova/blob/master/doc/source/contributor/develo… >> but we dont really adverstise that because none of the core maintainers >> use that any more and it has not been maintained >> for the better part of a decade. since we cant use this type of debugging >> with our ci system >> if we cant debug why a failure happens form the ci logs we add more logs >> because thyat will be useful for future us >> and operators in production. so our focus is on logging/monitoring for >> production rather then for developer eases >> simply because of the large techdebt that eventlet creates in this aready. >> >> i know nuetorn has had some better look with using debuggers then nova >> has, >> >> keystone/placmnet are eventlest free adn are just python web apps/rest >> apis so should "just work" >> in any python debugger. >> >> nova is prehaps the hardest to debug because of the legacy of the proejct >> so its not the best palce to start looking >> how this works. too be clear i have ran nova api, nova-conductor, nova >> scheduler and nova-compute all in 4 ide windows >> with 4 debuggers at once >> it was proably 3 year of working on openstack before i understood how to >> hack that ot work and since i learned >> how to properly debug without a debuggger, logs, code inspection, >> unit/fucntioal testing >> i have nver really need to do that again. >> >> >> gaining the ablity to run openstack in a debugger simply again, in a >> maintainable way is deffienly one of the reason im >> looking forward to removing eventlet but its still a lot of work. >> im not sure how helpful this was but this is proably as clsoe to a blog >> post as you will get. >> >> > >> > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote: >> > >> > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote: >> > > > How many more releases are you planning to include eventlet in >> > > > them? Seems they are removing any support for further development >> > > > according to their official website. So, is the community thinking >> > > > of any alternatives? >> > > [...] >> > > >> > > Just to expand on the first part of Sean's reply, most of what you >> > > want to know is probably answered here: >> > > >> > > >> https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html >> > > >> > > -- >> > > Jeremy Stanley >> > > >> >>

11 months, 4 weeks

Re: [eventlet-removal]When to drop eventlet support

by Sean Mooney

On 16/06/2025 15:33, Dmitriy Rabotyagov wrote: > Also let's keep in mind, that only nova (with placement) will spawn 64 > threads on such setup by default. yep see my other reply. you mixign up workers and eventlet threads. those are two very diffent things. we likely shoudl change the upstream default for workers to 1. we do tend to override that by defualt in installer tools. > > And then all really depends on set of services to launch on such setup. > > So from deployment tooling protective you have all required data to > rollout not instantly oom-ing setup at cost of amount of request > services can process in parallel. > > On Mon, 16 Jun 2025, 15:24 Dmitriy Rabotyagov, > <noonedeadpunk(a)gmail.com> wrote: > > In case you try to use a 32gb box with 16 cores as a controller > for OpenStack - it will blow off with default amount of workers > for wsgi and /or eventlet apps. > > While you can argue this should not be used as production setup, > this can be totally valid for sandboxes and we wanna provide > consistent and reliable behavior for users. > > But my argument was not in if/how we want to fine-tune > deployments, but also understand and provide means to define > what's needed as well as potential ability to revert in worst case > scenario as a temporary workaround. > So still some variables and logic would be introduced from what I > understand today. > > > On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote: > > > On 16/06/2025 13:27, Dmitriy Rabotyagov wrote: > > > > > > > > sayint its FUD is not helpful. > > > > we got a driect ask form operator and soem core to not > do a hard > > switch > > over. > > > > and while i wanted to only support one model for each > binary at a > > time > > we were sepcificly ask to make it configurable. > > > > > In the later case, your only available action is help > fixing > > bugs. It > > > is not up to the operators to double-guess what may or > may not > > work. > > > > correct we are not planning to document how to change > mode we were > > planning to only use this configuration in ci and > operator would be > > > > > > Well, we'd need to have that communicated so that deployment > toolings > > could adopt their setup to changes, as, for instance, in OSA > amount of > > eventlet workers are calculated based on the system facts, > so we'd > > need to change the logic and also suggest how users should > treat this > > new logic for their systems. > > why is OSA doing that at all today? > we generally don't recommend changing those values from the > default > unless you really know what your doing. > i don't think other installer do that. > tripleo, kolla-ansbile and our new golang based installer do > not, nor > does devstack so its surprising to me that OSA would change > such low > level values > by default. > > we will document any new config options we and and we are > documentation > how to tune the new options for thread pools but we do not expect > installation > tools to modify them by default. we are explicitly not making the > options based on the amount of resources on the host i.e. > dynamically > calculated based > on the number of CPU cores. > > for example we are explicitly setting the number of > scatter_gather > thread in the the dedicated thread pool to 5 > why its a nice small number that will work for most people out > of the box. > > can you adjust it, yes but it scale based on the number of > nova cells > you have an 99% wont have more then 5 cells. > > using information about the host where the API is deployed to > infer the > value of that would be incorrect. > > you can really only make an informed decision about how to > tune that > based on monitoring the usage of the pool. > > that how we expect most of the other tuning options to go as well. > > our defaults in nova tend to be higher then you would actually > need in a > real environment so while it may make sense to reduce > them we try to make sure the work out of the box for most people. > > gibi id building up > https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co… > > as part of nova move to encode this but our goal is that > deployment > tools shoudl not need to be modifyed to tune these > valued by defualt. > > > > > So it will be kinda documented in a way after all. > > > > > > told for a given release deploy this way. > > > > this is an internal impelation detail however we are not > prepared to > > deprecate usign eventlet until we are convicned > > > > that we can run properly without it. > > > > > For beginners, this would be a horrible nightmare if > default > > options > > > simply wouldn't work. We *must* ship OpenStack working > by default. > > no one is suggesting we do otherwise. > > > > > > Cheers, > > > > > > Thomas Goirand (zigo) > > > > > >

1 month, 3 weeks

[neutron] Disable fullstack CI job

by Rodolfo Alonso Hernandez

Hello Neutrinos: As commented yesterday in the Neutron meeting [1], the "fullstack" CI is blocking the eventlet removal. This CI FW is also blocking the SDK migration [2]. The refactor of this CI is now not possible due to time constraints. Because of this, we are going to disable it and migrate the tests implemented to other FWs, if possible. The coverage of this CI is important but limited. The advantage of this FW is the possibility of, for example, restarting services or agents; this is something achieved by other FWs like "tobiko". Another advantage is the possibility of testing with different configurations, but that could also be achieved by using the same CI job with different static configurations; it is also possible to change the API configuration and restart it using "whitebox-neutron-tempest-plugin". I've created a bug to track these efforts [3]. In the next few hours I'll send a patch to remove the "fullstack" FW from our CI. Regards. [1] https://meetings.opendev.org/meetings/networking/2025/networking.2025-06-24… [2]https://review.opendev.org/c/openstack/neutron/+/947851 [3]https://bugs.launchpad.net/neutron/+bug/2115327

1 month, 2 weeks

[neutron] Bug deputy report (week starting on February 3)

by Bernard Cafarelli

Hey neutrinos, here are the reported during the first week of February All bugs have a fix (merged in several cases) or assignee. Only pending one is the last one reported on neutron-vpnaas (but on EOL branch) Critical * [OVN] Neutron OVN metadata agent broken - https://bugs.launchpad.net/neutron/+bug/2097273 Fixed by https://review.opendev.org/c/openstack/neutron/+/940679 * networking-bagpipe CI is broken since linuxbridge driver removal - https://bugs.launchpad.net/neutron/+bug/2097367 Mentioned in last week team meeting, picked up by Lajos High * test_async_process_respawns_with_race_condition functional test fails randomly - https://bugs.launchpad.net/neutron/+bug/2097284 Recent new functional test failure - fixed with https://review.opendev.org/c/openstack/neutron/+/940687 * test_async_process_respawns functional test failing in fips functional job - https://bugs.launchpad.net/neutron/+bug/2095598 Another recent failure - fixed in the end with previous fix + https://review.opendev.org/c/openstack/neutron/+/939627 Medium * [eventlet-removal] Use ``ThreadPoolExecutor`` for ``periodics.PeriodicWorker`` - https://bugs.launchpad.net/neutron/+bug/2097257 Title says it all - reported and picked up by Rodolfo * devstack-tobiko-neutron failing in zed and 2023.1 with missing designate - https://bugs.launchpad.net/neutron/+bug/2097286 Tobiko job failures in unmaintained branches since job config cleanup - should be good with https://review.opendev.org/c/x/devstack-plugin-tobiko/+/940694 * [OVN] OVN metadata agent should build the response according to the content - https://bugs.launchpad.net/neutron/+bug/2097431 Fix when body response is empty with merged https://review.opendev.org/c/openstack/neutron/+/940679 Untriaged * No connection from outside host to VPNaaS host after upgrading from Ussuri to Victoria/Wallaby - https://bugs.launchpad.net/neutron/+bug/2097610 VPNaaS issue on EOL branch - I did not close immediately in case vpnaas experts have an idea -- Bernard Cafarelli

6 months

Re: [all][tc] Eventlet migration: Call to action

by smooney＠redhat.com

On Thu, 2024-05-16 at 10:27 +0200, Herve Beraud wrote: > Thanks Jay for this thread and for summarizing the T.C feeling. > My following sentences are not directly addressed to you (Jay), but to > others T.C members. > > Le mar. 14 mai 2024 à 18:28, Jay Faulkner <jay(a)gr-oss.io> a écrit : > > > Hi all, > > > > As you may have read in Goutham's TC summary, the existing eventlet > > goal[0] is unlikely to gain consensus to merge. This appears to be due to > > the difference in needs between OpenStack projects, and belief that it's > > unlikely a single solution would serve them all. > > > > I might agree with the fact it's unlikely a single solution would serve > them all. I was going to update some sentence to make this optionality more > stronger: > https://review.opendev.org/c/openstack/governance/+/902585/comment/bb363bda… > > Also, I was going to propose some adaptations to introduce futurist and > executors: > https://review.opendev.org/c/openstack/governance/+/902585/comment/6b23377c… speaking of executors for nova we have approved a specless bluepirnt to adress some of the low hanging fruit https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 which is implemented here https://review.opendev.org/q/topic:%22eventlet-removal-part-1%22 although i will be revising that at least once more before its ready to merge. the last two patches in the series remove eventlet.tpool and replace its use with futurist a thread pool executor instead for our specific usage https://review.opendev.org/c/openstack/nova/+/917962/4/nova/storage/rbd_uti… that's a relatively simple transformation. there are one our too outer uses of eventlet.tpool in nova and i will also adress the usage in the guestfs module but i will likely no touch its usage in the libvirt driver as we have a rather complex code dedicated to how we interact libvirt that need careful thought to untangle. > > But, rather than speaking of "a single solution", I'd suggest speaking > about "a solution by default". > > Indeed, many comments made in this proposal strongly adopt Nova's > perspective. > Nova is a major piece of Openstack, but Openstack is not only Nova. > By updating that sentence (see the previous links), we would make this > proposal optional for teams who have the capabilities to design their own > alternatives. Nova. > > Teams like Nova have resources that other teams may not have. yes and no. the nova teams capacity is fairly limited these days. we have between 5-10 active contributors depending on how you look at it. refining that by the people that understand our use of eventlet and can review the code and factor in have the time rewrite the relevnet code while being upstream consistently enought to actually make progress that probably means about 3-5 people could actually do this. at lesat 2 of those people are actively working on finisnhing multi cycle efforts and the rest including myself are heavily engaged in dowstream work or work in other projects that limits our upstream review and development time for at least the next 3-6 months. Nova is trying to be pragmatic that we do not have enough contributor to rewrite the part of nova that would require it to asyncio without heavily impacting the projects viablity. so we are looking at using futurist executors instead because its a lot less work to achieve the goal of not using eventlet while not prohibiting us form exploring asyncio or other options eventually if it makes sense. if we had to replace eventlet with asyncio im not confident we could complete that for nova in the next 12 -18 months. not without effectively stopping all other development. we just don't have the review or code writing capacity to do something that large and complex while actually ensuring we don't introduce regressions if we were doing anything else. > T.C members should consider this difference in resources between teams. > > T.C members are elected to adopt a global point of view, and not a team > based point of view: > https://governance.openstack.org/tc/reference/principles.html#openstack-fir… > > Else, without any default optional solution, teams with lower resources > will be deprived of any solutions... > > Teams should not consider the use of their surplus as legitimate when other > teams are deprived of what is necessary. > > As a community member, this is the kind of mindset that I expect from the > Openstack governance. > > > > Our current situation is: > > 1) Eventlet usage must be discontinued. The temporary maintainers are just > > that -- temporary -- and we should not expect eventlet to continue to be a > > valid path moving forward. > > 2) We have basic agreement that shared tooling, such as oslo libraries, > > should not dictate a specific threading model. > > > > If by "should not dictate a specific threading model", you mean that Oslo > forced the use of async/await, then I STRONGLY disagree about 2)... > Please show me where, in the proposed goal, oslo libraries dictate a > specific threading model? => > https://review.opendev.org/c/openstack/governance/+/902585 that more a refence ot not forcing async await rather then the internal impelmations. i.e. dont require your users to use asynio > > FYI the libs migration is detailed in the "How to migrate a library" > section (around line 646 of the proposed goal). > And a living example of how the libs would work is available at > https://github.com/4383/snippets/blob/main/python/facade/facade.py that snipit seams to assume its oke to create an asyncio eventlop on the fly in any geiven invocation which is proably going to be too slow for many usecases i.e. if oslo log did that every time we logged that would be a problem. i have not messuered the startup time of the asynio eventloop but i expect its more expensive then a trival python function call. if that loop was reused between calls to the lib or even kicked out into its own thread perhaps with a graceful shutdown if noting is run for an extended period of time like privsep, then i can see that variation fo the facade pattern working. we ould have to mussure the overhad of _iter_coroutine https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_runnin… does appear to be thread safe so in principal it should not cause any cross thread interactions. that was one of my concerns when lookign at the snipit initally. > > To conclude, is it still worth it to spend time updating this proposal > again or is your decision definitive? >

1 year, 2 months

openstack-discuss search results for query "#eventlet-removal"