Search results for "#eventlet-removal" - openstack-discuss

#eventlet-removal Thank you for 2024

by Herve Beraud

You are no doubt aware that we are currently facing a real challenge. We are living in a complex situation where we have to engage resources to exit from a problematic technology, eventlet, without especially gaining new features. This migration, although without immediate direct functional value, represente for us a considerable investment in time and resources. But such context is not far different from the Python 2/Python 3 migration. Remember that era. The core Python maintainers found themselves in a difficult situation: continue to maintain their code on Python 2, which had become insecure and unsupported, or go through the effort of completely migrating their codebase and toolchain to Python 3. All the community had to make a choice. Third-party libraries, frameworks were also on the edge case, to be upgraded or replaced with Python 3 compatible equivalents. At the time of the decision, these efforts appeared to represent a "non-value-added burden". These efforts did not bring any new functionality, did not respond to any new customer need, and even slowed down other more innovative developments. The Python maintainers could have chosen the easy way. They could have chosen to continue to support Python 2 again for years, they could have chosen to live in statu-quo, but that's not what happened [1]. They chose to fight the problem and they chose to engage strong efforts in this migration. They ensured the security and the perenniality of the Python programming language. Once the migration was completed, the benefits quickly became apparent. The innovation cadence increased. The community united behind one uniq major version. Better performances. More innovations. And last but not the least, a simpler life for the distros maintainers and for end-users, and so a reinforced confidence in Python by our industry. Python is now the number 1 technology, leaving Java and C in the dust [2]. This is the lesson we must learn. Necessary actions without immediate direct functional value can ultimately hide treasures of innovation. Engagement and dedication is the key to a sustainable future. It guarantees the long-term viability of a project, reduces risks, and opens the way to faster and more peaceful innovation. Today, I can tell you that I'm proud that our community engaged the same way. I'm proud to see all the milestones we reached during this exciting year. I'm proud to observe so much engagement coming from every side, from everybody, and from all our companies. That's a strong signal of trust from our industry. Our companies support this initiative, our companies support open source, our companies invest in the future of OpenStack. On behalf of this community initiative, let me thank you very much for all your efforts and engagements in 2024. Let me wish you happy holidays and enjoy your loved ones. [1] https://devguide.python.org/versions/ [2] https://www.tiobe.com/tiobe-index/ -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

7 months, 3 weeks

Re: [eventlet-removal]When to drop eventlet support

by Dmitriy Rabotyagov

Also let's keep in mind, that only nova (with placement) will spawn 64 threads on such setup by default. And then all really depends on set of services to launch on such setup. So from deployment tooling protective you have all required data to rollout not instantly oom-ing setup at cost of amount of request services can process in parallel. On Mon, 16 Jun 2025, 15:24 Dmitriy Rabotyagov, <noonedeadpunk(a)gmail.com> wrote: > In case you try to use a 32gb box with 16 cores as a controller for > OpenStack - it will blow off with default amount of workers for wsgi and > /or eventlet apps. > > While you can argue this should not be used as production setup, this can > be totally valid for sandboxes and we wanna provide consistent and reliable > behavior for users. > > But my argument was not in if/how we want to fine-tune deployments, but > also understand and provide means to define what's needed as well as > potential ability to revert in worst case scenario as a temporary > workaround. > So still some variables and logic would be introduced from what I > understand today. > > > On Mon, 16 Jun 2025, 14:43 Sean Mooney, <smooney(a)redhat.com> wrote: > >> >> On 16/06/2025 13:27, Dmitriy Rabotyagov wrote: >> > >> > >> > >> > sayint its FUD is not helpful. >> > >> > we got a driect ask form operator and soem core to not do a hard >> > switch >> > over. >> > >> > and while i wanted to only support one model for each binary at a >> > time >> > we were sepcificly ask to make it configurable. >> > >> > > In the later case, your only available action is help fixing >> > bugs. It >> > > is not up to the operators to double-guess what may or may not >> > work. >> > >> > correct we are not planning to document how to change mode we were >> > planning to only use this configuration in ci and operator would be >> > >> > >> > Well, we'd need to have that communicated so that deployment toolings >> > could adopt their setup to changes, as, for instance, in OSA amount of >> > eventlet workers are calculated based on the system facts, so we'd >> > need to change the logic and also suggest how users should treat this >> > new logic for their systems. >> >> why is OSA doing that at all today? >> we generally don't recommend changing those values from the default >> unless you really know what your doing. >> i don't think other installer do that. >> tripleo, kolla-ansbile and our new golang based installer do not, nor >> does devstack so its surprising to me that OSA would change such low >> level values >> by default. >> >> we will document any new config options we and and we are documentation >> how to tune the new options for thread pools but we do not expect >> installation >> tools to modify them by default. we are explicitly not making the >> options based on the amount of resources on the host i.e. dynamically >> calculated based >> on the number of CPU cores. >> >> for example we are explicitly setting the number of scatter_gather >> thread in the the dedicated thread pool to 5 >> why its a nice small number that will work for most people out of the box. >> >> can you adjust it, yes but it scale based on the number of nova cells >> you have an 99% wont have more then 5 cells. >> >> using information about the host where the API is deployed to infer the >> value of that would be incorrect. >> >> you can really only make an informed decision about how to tune that >> based on monitoring the usage of the pool. >> >> that how we expect most of the other tuning options to go as well. >> >> our defaults in nova tend to be higher then you would actually need in a >> real environment so while it may make sense to reduce >> them we try to make sure the work out of the box for most people. >> >> gibi id building up >> >> https://review.opendev.org/c/openstack/nova/+/949364/13/doc/source/admin/co… >> >> as part of nova move to encode this but our goal is that deployment >> tools shoudl not need to be modifyed to tune these >> valued by defualt. >> >> > >> > So it will be kinda documented in a way after all. >> > >> > >> > told for a given release deploy this way. >> > >> > this is an internal impelation detail however we are not prepared to >> > deprecate usign eventlet until we are convicned >> > >> > that we can run properly without it. >> > >> > > For beginners, this would be a horrible nightmare if default >> > options >> > > simply wouldn't work. We *must* ship OpenStack working by default. >> > no one is suggesting we do otherwise. >> > > >> > > Cheers, >> > > >> > > Thomas Goirand (zigo) >> > > >> > >> >>

1 month, 3 weeks

[tc][ptg] OpenStack Technical Committee 2025.2 Flamingo PTG summary

by Goutham Pacha Ravi

Hello Stackers, Here’s a summary of the OpenStack Technical Committee’s discussions at the 2025.2 “Flamingo” Virtual PTG held April 7–11, 2025. Find detailed notes in our discussion etherpad [1] and recordings on YouTube [2]. Feedback is welcome on this mailing list or in OFTC’s `#openstack-tc` IRC channel, perhaps during our weekly meetings [3]. == Topic: TC Retrospective == Discussion: - We reviewed the TC’s effectiveness in addressing issues raised during the cycle, including leadership gaps, licensing concerns, project guide updates, and communication practices. - Fewer teams were leaderless following the 2025.2 election cycle—thanks to better outreach and clearer election guidance. - There was consensus that the TC guideline discouraging bare “recheck” comments and the need to address persistent gate issues is now better understood. - Licensing FAQs [4] appear outdated. We noted that the TC can escalate licensing questions to the OpenInfra Foundation’s Legal team and should monitor the `legal-discuss` list more actively. - Teams encouraging non-core contributors to become PTLs was praised as a way to grow the contributor base through mentorship. Action Items: - Update the Project Team Guide, removing obsolete sections (e.g., sprint design/process). - Update licensing references across the Project Team Guide and TC governance site. - Encourage public communication, regular meetings, and the use of team-specific IRC or Matrix channels. == Topic: Goals Retrospective == Discussion: - We reviewed the status of ongoing community-wide goals, including progress, blockers, and next steps. - Migrate CI Jobs to Ubuntu Noble [5] - The goal is complete. Teams should now remove OS version pins in their repositories. - Eventlet Migration [6] - Refer to Hervé’s PTG summary [7] for updates. - Privsep Migration [8] - Progress is slow; some minor activity exists in Gerrit, but tracking doesn’t reflect it. - The Oslo team will de-prioritize removing eventlet from `oslo-rootwrap`, as it’s expected to be retired. - Clear guidance on secure, performant `privsep` usage is needed. - FIPS Compliance [9] - CentOS Stream 10 requires x86_64-v3, unsupported by our nodepool providers. - This hardware/runtime mismatch complicates long-term testing with CS9 or Ubuntu LTS. - Alternatives discussed: Debian, Rocky Linux (but Rocky may inherit CS10's hardware constraints). - Red Hat is evaluating a `paramiko` replacement this cycle. - Consistent and Secure RBAC [10] - Progress tracked [here](https://etherpad.opendev.org/p/rbac-goal-tracking#L14). - Phase 1 (personas + drop system scope): 16 projects complete. - Phase 2 (service role): 3 projects. - Phase 3 (manager role): 3 projects. - Ghanshyam Mann (gmann) continues to chair bi-weekly IRC meetings. Action Items: - Rodolfo Alonso Hernandez (ralonsoh) to send a status update on `privsep` and flag possible `rootwrap` deprecation. - Ade lee (ade_lee) to update FIPS goal status. - SRBAC liaisons must attend bi-weekly meetings and track project progress. - Publish guidance on secure, performant `privsep` usage. == Topic: Revisiting “Unmaintained” Branches == Discussion: - The TC discussed the burden and ambiguity around maintaining aging branches. - Project teams often lack resources or interest to maintain older branches; the Release team has pursued mass EOL as a workaround. - We agreed to reduce scope and prioritize core goals (e.g., eventlet removal, CI stability) rather than preserve old branches without active interest. - TC supports giving `openstack-unmaintained-core` full responsibility for EOL decisions. We may seek volunteers earlier in the branch lifecycle. - Broken jobs in unmaintained branches flood Zuul; fixing in-repo configs may not be viable. OpenDev may opt to ignore failing jobs externally. - There’s an unregistered, unlogged `#openstack-unmaintained` IRC channel on OFTC. Action Items: - Document responsibilities of unmaintained branch liaisons and `openstack-unmaintained-core`. - Propose a new process: `openstack-unmaintained-core` sets deadlines per cycle to solicit maintainers for aging branches. If no one steps up, proceed to EOL. == Topic: TC and Community Leaders Forum == Discussion: - Governance proposal under review to expand the scope of the VMT to cover all OpenStack project teams [11]. - The TC must ensure teams prioritize security. Poor handling of vulnerabilities harms community trust and ethical responsibility. - Ideas were proposed to support the VMT, including a pool of core security contacts from contributing organizations. - Projects were reminded to review community goals and prioritize them. Adding goal timelines and cross-project prioritization was suggested. - Teams should use Stackalytics and Bitergia dashboards to track activity and identify areas needing improvement (e.g., review velocity, core participation). Action Items: - Teams must audit their Launchpad “Bug Sharing” settings and confirm their `coresec` groups contain responsible, active contributors. - PTLs and Security Liaisons should ensure they can manage the `coresec` group or delegate access. - Audit deliverables against [Security Process Requirements][12]. - Benchmark project health using Stackalytics/Bitergia. Reinforce OpenStack code review best practices [13][14]. == Topic: Finding Contributors in Some Projects is Hard == Discussion: - QA, Requirements, and Release Management teams remain understaffed. - Despite upstream investment opportunities for 2025 [15], contributor interest is low. - This is a “Tragedy of the Commons” issue: shared work without clear ownership is often deprioritized. - PTG scheduling didn’t support sufficient cross-project collaboration. - The TC was encouraged to reset perfectionist expectations—approve good-enough changes, and avoid expanding review scope unnecessarily. Action Items: - Explore QA Liaisons: trusted contributors mentored into maintaining QA repos. - Avoid nitpicking or scope creep during code reviews. - Promote and participate in the contributor survey [16]. - Document expectations and best practices in the Project Team Guide. == Topic: Splintered Conversations & Communication Barriers == Discussion: - Operators report more engagement on Reddit, Kubernetes Slack, Discord, etc., than on `openstack-discuss` or OFTC IRC. - Some organizations block Matrix due to regulatory compliance, requiring platforms with retention policies. - The OpenInfra Board will not mandate platforms—projects must choose what suits them. - IRC onboarding is hard for newcomers. - Zuul contributors reported smooth onboarding to Matrix. Its e2e encryption may make it more acceptable for compliance-sensitive orgs. - The conversation continued in the `os-operators` PTG, where there was interest in trying Matrix and Meetpad for better engagement. Action Items: - Explore use of a hosted Matrix homeserver by OpenDev as a short-term workaround for those unable to use IRC. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://etherpad.opendev.org/p/r.51cb4e74dd9980932d32e9a9f4e8a0ce (OS TC PTG Etherpad) [2] https://www.youtube.com/playlist?list=PLhwOhbQKWT7XYzqrx21j1uizxNxnbDf5Q (OS TC PTG Recordings) [3] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee (OS TC Weekly Meetings) [4] https://wiki.openstack.org/wiki/LegalIssuesFAQ (OpenStack Licensing FAQs) [5] https://governance.openstack.org/tc/goals/completed/2025.1/migrate-ci-jobs-… (Migrate to Noble goal) [6] https://governance.openstack.org/tc/goals/selected/remove-eventlet.html (Eventlet removal goal) [7] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… (Eventlet PTG Summary) [8] https://governance.openstack.org/tc/goals/selected/migrate-to-privsep.html (Privsep migration goal) [9] https://governance.openstack.org/tc/goals/selected/fips.html (FIPS compliance goal) [10] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… (SRBAC goal) [11] https://review.opendev.org/c/openstack/governance/+/944817 (TC VMT Resolution) [12] https://security.openstack.org/repos-overseen.html#requirements (Security requirements for OpenStack Projects) [13] https://docs.openstack.org/project-team-guide/review-the-openstack-way.html (Code Reviews in the OpenStack Way) [14] https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatt… (Code Review Antipatterns) [15] https://governance.openstack.org/tc//reference/upstream-investment-opportun… (OpenStack Investment Opportunities) [16] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… (Bridging the Gap)

3 months, 3 weeks

[nova][ptg] Nova sessions on Wednesday

by Sylvain Bauza

Hi, Given most of the nova cores will attend the eventlet-removal session at 1300UTC, we'll defer the nova sessions to start at 1400UTC instead of 1300UTC. Please stay online in #openstack-nova IRC channel or please take a look at https://ptg.opendev.org/ptg.html to see the exact time when we spawn again the nova conversations. Thanks, -Sylvain (who just unbooked the austin room for Wed-B1)

9 months, 3 weeks

[mistral] meetings during next PTG

by Arnaud Morin

Hey all, Next PTG will take place between April 7 to 10. Is there anyone willing to talk about mistral developement during this PTG? So far, mistral has not evolved that much, but maybe someone is having ideas about it? Also, there is this eventlet-removal topic that could be discussed. Let me know, Cheers, Arnaud.

6 months, 2 weeks

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.2/R-12)

by Goutham Pacha Ravi

Hello Stackers, We're officially in the second half of the OpenStack 2025.2 "Flamingo" release cycle [1]. Starting on 2025-08-06, election officials will begin merging nominations for Project Team Leads for the 2026.1 release cycle and to fill four seats on the OpenStack Technical Committee. If you're interested in leading a project or joining the OpenStack TC, I'd encourage you to submit a nomination through the openstack/election repository [2]. The Call for Proposals soliciting project team updates and Forum sessions at the OpenInfra Summit (17–19 October 2025) in Paris-Saclay will be closing on 2025-07-08 at 23:45 PST [3]. The schedule for the event is live and contains some exciting material [4]. In the past week, the OpenStack Technical Committee didn't merge any significant governance changes. However, some proposals are open for the community's review. === Weekly Meeting === The last weekly meeting of the OpenStack TC was held simultaneously on Video (Meetpad) and IRC (OFTC's #openstack-tc channel) on 2025-07-01. The recording of the meeting was posted to the TC's YouTube channel [5], and meeting notes were logged on IRC [6]. The meeting was brief yet efficient, covering updates on various ongoing initiatives. TC members were glad to see the smooth implementation of the Developer Certificate of Origin (DCO) enforcement, with minimal issues reported and proactive steps taken to manage tooling changes. Key action items from the previous week, such as addressing inactive projects and the Eventlet deprecation, were discussed. We noted progress on Cyborg's potential revitalization through new and returning maintainers. We discussed the critical and impending bug in PBR [7] due to upstream changes in Setuptools. The OpenStack user survey remains open until August [8], and the TC would like broader community participation. Lastly, we made plans for a "meet-and-greet" forum session with the OpenStack TC at the upcoming summit. This would be a valuable opportunity for community engagement and discussion of TC responsibilities. We do hope to have at least half of the current TC at the summit. The next meeting of the OpenStack TC is on 2025-07-07, and will be hosted on the #openstack-tc channel on OFTC at 1700 UTC. Please find the agenda and other details on the meeting's wiki page [9]. I hope you'll be able to join us there! === Governance Proposals === ==== Merged ==== - Remove mentions of CLA in licensing guide | https://review.opendev.org/c/openstack/governance/+/953843 ==== Open for review ==== - os-test-images never releases | https://review.opendev.org/c/openstack/governance/+/954249 - Add service-types-authority to SDK deliverables | https://review.opendev.org/c/openstack/governance/+/953548 - Deprecate shade, os-client-config | https://review.opendev.org/c/openstack/governance/+/953549 - Remove Monasca from active project | https://review.opendev.org/c/openstack/governance/+/953671 - Make Eventlet removal deadlines more acceptable for operators | https://review.opendev.org/c/openstack/governance/+/952903 - Require declaration of affiliation from TC Candidates | https://review.opendev.org/c/openstack/governance/+/949432 === Upcoming Events === - 2025-07-08: OpenInfra Board meeting: https://board.openinfra.org/ - 2025-07-19: OpenInfra Days, Indonesia: https://2025.openinfra.id/ - 2025-07-26: OpenInfra Days, Vietnam: https://www.vietopeninfra.org/void2025 - 2025-08-06: Nominations open for OpenStack elections: https://governance.openstack.org/election/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.2 "Flamingo" Release Schedule: https://releases.openstack.org/flamingo/schedule.html [2] Submit a PTL/TC candidacy: https://governance.openstack.org/election/#how-to-submit-a-candidacy [3] CFP for Forums/Project Updates: https://cfp.openstack.org/app/2025summit [4] OpenInfra Summit Schedule: https://summit2025.openinfra.org/a/schedule [5] TC Meeting Video Recording, 2025-07-01: https://youtu.be/eJ03g0w8Ers [6] TC Meeting IRC Log 2025-07-01: https://meetings.opendev.org/meetings/tc/2025/tc.2025-07-01-17.00.html [7] pbr setuptools incompatibility: https://bugs.launchpad.net/pbr/+bug/2107732 [8] Take the OpenStack user survey: https://www.openstack.org/user-survey [9] TC Meeting Agenda, 2025-07-08: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 month

Re: [oslo] Remove eventlet from openstack.

by smooney＠redhat.com

replacing oslo.service was not in the set of thing we were condierign doing in nova this cycle but if we compelte the ohter work and i finish the other feature im ment to work on i might try swaping in Cotyledon if anyone trys that for any other service or feels like trying it in nova the i owuld be happy to review to see what that looks like. looking at the gaps im not sure how much of them actully matter at the end of the day i.e. we planned ot remoe the eventlet backdoor debuging supprot with the removal of eventlet anyway so thet fact Cotyledon does not hook that up is fine. similarly we talked about removing the standaloen wsgi console script for nova-api or replacing it with something else like flask ectra so the wsgi supprot might not be an issue either. we will just have to asses those gaps when it comes to proting and if there is an upgrade impact that we can or cannot accept. teh fact that ceilometer and aodh use Cotyledon is at least encuraging espically since that implies the distor packageing shoudl already be in place but i just wanted ot make it clear i was not focusing on the governace question but also the technical gaps i was expeict that oslo.service would not be released as part of the eventlet removal and instead would have been ported to not depend on eventlet. i.e. have oslo.service perodici tasks just delegate to futurist providing the same api with a diffent backend. so i was kidn of surprised to see this topic come up. ill see if i can find time to do a poc but it wont happen until at least late july or august based on the other thing on my plate so if other do have a poc to share of portign ironic or neuton or cinder ectra that might fill in some of the gaps. On Tue, 2024-06-11 at 23:11 +0100, smooney(a)redhat.com wrote: > On Tue, 2024-06-11 at 22:36 +0200, Andriy Kurilin wrote: > > Hi! > > > > вт, 11 черв. 2024 р. о 16:26 <smooney(a)redhat.com> пише: > > > > > On Tue, 2024-06-11 at 14:09 +0000, Dmitry Tantsur wrote: > > > > Hi, > > > > > > > > On 6/10/24 10:30 PM, Daniel Bengtsson wrote: > > > > > > > > > > > > > > > On 2024-06-10 16:28, Dmitry Tantsur wrote: > > > > > > If we go with this proposal (which sounds reasonable), we need to > > > > > > carefully look at cotyledon's maintenance. It seems to be on life > > > > > > support by only one volunteer: > > > > > > https://github.com/sileht/cotyledon/commits/main/ > > > > > You're right but he is pretty active and I don't think it's a big > > > issue. > > > > > > > > How do you judge that? Herve mentioned a PR merging quickly, but that's > > > > not a perfect criteria. I have a pet project (rust-openstack) where I > > > > also tend to merge PRs quickly. Except when I'm on a vacation or very > > > > busy at work, then I can forget about PRs for a month or more. > > > > > > > > Note that I'm not saying "let's not use it". I think a reasonable next > > > > step, if we decide to research this part, would be to get in touch with > > > > the maintainer, understand their plans and their position on adding more > > > > maintainers in the future. > > > > > > i think in one of the other replies they mentioned propsoing it as a > > > deliverbale > > > under the oslo project so if that happens i think the maintainer ship and > > > co installablity (use of upper constratits) issues as well as goverance > > > issues > > > go away > > > > > > > Why do we still rely on SQLAlchemy and do not request Michael Bayer to move > > the library under OpenStack governance? Most contributions come from a > > single maintainer. The project already uses Gerrit, so the contribution > > flow will be similar, right? > > > > Also, do you believe that moving projects under OpenStack governance > > magically increases project aliveness? > > Let's discuss the "ldappool" library. It is under OpenStack governance and > > is required for LDAP integration in Keystone. Is it alive? The last release > > was on February 26, 2021. This release includes a bug: the usage of a > > dependency that is missed in requirements.txt - the "six" library. > > The fix waited for two years to be merged > > https://review.opendev.org/c/openstack/ldappool/+/805495 , and there is no > > new release with it. > > Hm... So, magic OpenStack governance did not help?! How did this happen? > > > > Sorry for these sarcastic questions. I only tried to say that independently > > of project governance, the project may die and become unmaintained. The > > only valuable thing is the contributors. > > Asking a project maintainer to move from "his" development flow to work > > under OpenStack governance before reducing "his real maintainer's costs" > > sounds strange. > > If openstackers start actively contributing to Cotyledon, moving under > > OpenStack governance as Cotyledon community decision sounds reasonable to > > me. Otherwise, it doesn't sound nice. It is not "a chicken&egg problem"; > > the primary action should be obvious. > > its not about being in or out of openstack or opendev governance. > when i looked at the repo and swap it was using different build tooling > was not using upper constraint and may or may not be co installable. > > for practial pourpuses we would need to make depend-on work in ci which > means usign the github zuul integration and ensuring that devstack libs_form_git funcitionatliy > works to be able to test changes before they merge. that woudl be simpler if it was hosted on opendev > but its not a blocker. > > cotyledon is using pytest as a test runner instead of stestr so im not sure if that > will integrate well with our ci tooling, i.e. we wont have subunit output to render test report for. but that porbaly > less problmatic then if it was actully using pytest as the test framework. > pytest can produce some nice html report but form normal openstack contibutor point > of view a proejct that is not aligned to the PTI will add a slight learing curve especially if > contirbutions is via github prs. > > there is enough divergance form what we normally use in the rest of openstack > to potentially be problemaitc in the furutre. for exmaple its not using oslo.logging for logging, > its using the pythong logging framework directly so we would need to ensure that it integrated > properly with oslo.logging and our exiting log cofniguration that provides. > > on the doc front it has very little docs in tree > and doees not appear to have a 1:1 mapping to the apis provided by oslo serviece. > > so my initial read is that it probably isn't a suitable drop in replacement in its current state. > > as oslo.service is a very core part of the services and we expect things like GMR ectra to ingenerate with > specific behavior related to the use of sig_user2 so we need to evaluate how Cotyledon is > operating system singles given the libary claims > > "Reload API (on SIGHUP) hooks work in case of you don't want to restarting children > A separated API for children process termination and for master process termination > " > > i think this is the relevnet code so it looks like it does not touch sig_user2 > https://github.com/sileht/cotyledon/blob/main/cotyledon/_service.py#L221-L2… > so the GMR aspect is proably fine > > what im less sure about is Cotyledon states it does not > > " > facilitate the creation of wsgi application (sockets sharing between parent and children process). Because too many > wsgi > webserver already exists. > " > > however nova does use that functionality in oslo.service > https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py > so that is a potentially a blocker for nova to use it unless we rewirte how our wsgi applicaiton works. > > for what its worth iwould nto be agaissnt replaceign our current usage with flask or similar but > a project that was built to replce oslo.service for celimiter is not nesssiasarly a good fit for > all other openstack porjects that use oslo.service. > > so to me Cotyledon has a lot of open question that woudl need to be answered not related to the > governance > > > > > > > > > > > > Dmitry > > > > > > > > > > > > > > -- > > > > > Daniel Bengtsson > > > > > Software Engineer > > > > > > > > > > > > > > > > > >

1 year, 2 months

Re: debugging eventlet services in vscode.

by Herve Beraud

Hey Sean, Thanks for sharing it with us. Do you want to add it in a more formal way in our toolbox/materials related to Eventlet? Maybe in a dedicated section inside https://wiki.openstack.org/wiki/Eventlet-removal or in a parallel document linked into the wiki? As you want. Le mar. 5 nov. 2024 à 02:37, Sean Mooney <smooney(a)redhat.com> a écrit : > > hi folks i recently had to debug a problematic interaction between > eventlet and a native thread in watcher. > while i probably could have printed my way though that eventually i > decided to try and get debugging working > in vscode instead. > > > in this case i happened to be looking at watcher but this should apply > to any eventlet service installed with devstack. > > > in my .vscode directoy i created a launch.json that looks like this. > > { > // Use IntelliSense to learn about possible attributes. > // Hover to view descriptions of existing attributes. > // For more information, visit: > https://go.microsoft.com/fwlink/?linkid=830387 > "version": "0.2.0", > "configurations": [ > { > "name": "Python Debugger: watcher-applier", > "type": "debugpy", > "request": "launch", > "program": "/opt/stack/data/venv/bin/watcher-applier", > "console": "integratedTerminal", > "args": [ > "--config-file", > "/etc/watcher/watcher.conf" > ], > "gevent": true, > "env" : { > "PYTEST_CURRENT_TEST": "true" > } > }, > { > "name": "Python Debugger: watcher-decision-engine", > "type": "debugpy", > "request": "launch", > "program": "/opt/stack/data/venv/bin/watcher-decision-engine", > "console": "integratedTerminal", > "args": [ > "--config-file", > "/etc/watcher/watcher.conf" > ], > "gevent": true, > "env" : { > "PYTEST_CURRENT_TEST": "true" > } > } > ] > } > > > there are a few odd things about this config. > > first the program i am launching is the console script run in the > devstack@* systemd service file. > you can find what that is using systemctl show devstack@<name> > to make this work without conflict do `sudo systemctl stop > devstack@<name>` first. > after your done debugging you can just start the systemd service again. > > second when using vscode to view the source of a project select the > devstack global virtual env as the project virtual env. > > third im telling vscode to enable gevent support. > now this will raise a warning/error as gevent is not installed but you > can ignore that. > for those of you not familiar with gevent its a fork of eventlet that > actually has much better integration > with debuggers due to its wider usage outside of openstack. > eventlet and gevent both use greenlet and the eventlet implementation is > close enough that when you enable gevent debugging > it mostly works. > > forth to fix the "mostly" part, specifically the fact that dajngo will > explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal > env. > > PYTEST_CURRENT_TEST is an undocumented environment variable intended > only for unit testing that is check by eventlet > > https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… > when we set that to true we skip this `isinstance(o, rlock_type)` check > > https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… > which crashes the debugpy debugger because it tries to load setting form > the danjgo module which get loaded into > the python interpreter as a side effect of debugpy. > > i have not tried this on other project yet but it enables me to single > step debug, run to breakpoints and many of the normal > debug action like using the debug console to inspect the current > function scope. > all of the above worked without any code change to watcher or running in > a specal debug mode in the service code. > > we occasionally get questions on this list about how to debug openstack > services and i usually respond with > > "its possible to do, is often fragile and need a lot of context to do > right" > > in this case it seam to work reasonable ok in the limited usage i have > made of it so far. > given the current incantation is pretty simple and non invasive i > decided to share this > in case others find it useful. > > keep in mind however most service are distributed systems so if you > leave it stopped > on a break point for 5 mins while you inspect things, what ever you were > originally doing will probably time out and > it might fail in odd ways but for local development that may still be > fine. please don't try this in production. > > i have not tried this with a wsgi app. that probably will work fine with > the eventlet wsgi web-server via the console scripts > > i.e. nova-api binary but probably wont work for uwsig/Apache. > > > regards > > sean > > > > > > > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

9 months, 1 week

sean speak and you! was Re: Eventlet and debugging

by smooney＠redhat.com

On Thu, 2024-08-15 at 17:36 +1000, Mike Carden wrote: > In almost a decade of reading smooney(a)redhat.com's Lysdexic posts, this is > the longest one I have ever read. No criticism intended at all - none > whatsoever. > > BUT... I am still absolutely Gobsmacked that anyone can possibly write even > vaguely working code, if their written English communications are so > comprehensively broken. How does anyone get away with ignoring Spelling and > Grammar in Code? > > I love your work Sean, but I do struggle to understand almost everything > you type. no worries short answer is i have more tooling to help when im writing code if im not in a rush or its a more formal topic i heavily rely on things like grammerly or spellcheckers to fix what i typed. as my typeing speed has increased the things i used to only do when writing i.e. swapping/skipping letters or writing a word other then the one i intended have unfortunately increased as i am now more or less using muscel memory for typing and that allow my dyslxia to propagate to my typing in a way that does not otherwise happen. when im coding i actually type much slower as i am mentally thinking more about the code and my ide will also complain at me. i even added codespell checking to nova recently with pre-commit to prevent future typos form being introduced to the codebase as they do get missed in review. tools like codepoilt can also help. when i use it is more or less like auto correct/spellchecking on steroids then actually for code generation. just be careful if you start a comment with NOTE(sean-k-mooney): i have seen it emulate how i write comment including typos before :) you hopefully wont have that problem but i have started to use it to help me write comments because its mostly less error prone then when is entrily write it by hand. in this case the main reason for all the typos is it was almost midnight and i was tired. the more tiered i am the more this happens, the same if i am distracted. effectively the more i rely on my touch typing ability and dont look a the keyboard the more mistakes i will make as when i read back what i typed I'm not actually reading. i think i am but what I'm actually doing is skiming the text and filling in the blanks form my working memory and not seeing what i typed. when it comes to code that is mitigated because i see the code in my editor, then i see it presented differently wiht git diff and finally after i push it i open it in gerrit and see it presented differently again there. that 3 phase reivew has enough visual different that i will actully read the text instead of rememebering it. the fact that copilot can help mask my dyslxia is actully the main reaon i didnt cancel my yearly description because i does help with that alot. i proably should pay for grammerly premium however. especially since if im not in a rush i used it to write all my specs after the initially draft. when i don't people have issues comprehending them :) and rightly so i sometime wonder how i typed what i typed and didnt notice :) if anyone has recommendation for email clients or plugins that can help let me know. i have been holding of actually using an IDE/Emacs for email but maybe i should. currently i use evolution but i used Thunderbird for years. in case its not obvious i spend about 15 mins actually fixing the typos in this email and im sure i have missed some. i find that incredible tiring but im always looking for ways to make my written communication better, so i welcome any suggestions. > > -- > MC > > > > On Thu, 15 Aug 2024 at 08:07, <smooney(a)redhat.com> wrote: > > > On Thu, 2024-08-15 at 01:49 +0530, engineer2024 wrote: > > > Thanks to both of you for the response . I try to insert pdb breakpoint > > at > > > some point in the code files and start the service. Then I issue a nova > > > command to check how the flow works. From then on I use the pdb commands > > to > > > check the function calls. This to me gives a very convenient way of > > > learning the code connections, since I don't have an idea of how these > > > functions and modules r related. U people as developers have a design to > > > start with and work on from there. So u may know the purpose of each > > > function, class, etc. > > > > > > Since nova api doesn't support pdb becos of eventlet design, I learned to > > > print those objects and find their information. > > > > > > But with pdb, it's a smooth flow to inspect the class attributes, their > > > functions, arguments, types etc. I can even test those objects the python > > > way since pdb supports python prompt as well. Of course I do all this on > > > dev systems though... > > > > this is going to be a long responce and its a bit rambely but im goign to > > provide > > you with the context and poitn er try and figure out how to do this for > > your self. > > > > first thing to note is while its possibel it not trivial and the payoff is > > proably not > > worth it in the long run. we do not have up to day docs for how to do this > > because > > this is not how peole learn, contibute to ro develop nova in genral. > > > > > > nova does have a remote debug facilitate that is sepreate for the eventlet > > backdor to enable > > you to use an ide to debug remotely. > > that was added around 2012 and has worked on an off to vering degrees > > since then. > > > > the first thing to understand about nova and several other project is that > > they are a distibuted system > > i.e. the applciaiton that compise a given service like nova run in > > multiple service which are interconnected > > vai a message bus with in a service and rest apis are exposed for inter > > service comunication. > > > > when you enable a debugger and stop on a breakpoitn only one of the > > process is stop and the other contiue > > to exsculte which means that RPC calls, heathbeats and other asynconos > > tasks can and will timeout and fail. > > > > you cannot debug a distibuted system in teh same way as you woudl a simple > > applction where all state > > is executed in a signle main loop. > > > > the act of observing the internal behavior of the nova api or nova-comptue > > agent will change its behavior. > > > > nova has suppoort for remote debugging that was added about 12 years ago > > https://wiki.openstack.org/wiki/Nova/RemoteDebugging > > https://github.com/openstack/nova/blob/master/nova/debugger.py > > > > with that supprot you can use an idea like pycharm to connect to the > > remote debugger and attempt to > > debug a process. > > > > im currently propsoing removing that as part of the removal of eventlet > > partly because it shoudl not be needed > > when we dont have enventlet and mainly because it didnt work very well > > when i last used it. > > > > some ides have supprot fo gevent > > > > gevent is a fork of eventlet which was forked form some of hte libiares in > > stackless python > > eventlet converts a single thread python interperater in a cooperative > > concurent multi userland process. > > > > what does that mean , each python function is not operating in a seperate > > os stackframe, it has heap allcoated > > stack frames that allow context swtichs between userland thread at any > > tiem a funnction woudl do blocking io > > > > the reason single step debugging does not work well with gdb is that when > > you single step the eventlet engine > > is often going to end up context switching into the midel of another > > function. > > > > to work around this you just set breakpoint at the line you want to go to > > next and use continue instead fo single step > > > > one of the rules when using eventlet is that if your going to monkey > > patch you need to monkey patch everyting early > > > > in addtion too the remote debuging facilites we can disabl mokey patching > > entirly. > > you can do that by settign OS_NOVA_DISABLE_EVENTLET_PATCHING=1 > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L87 > > > > this will entirly break nova-compute and several other nova process as > > they have > > effectivly infinity loops to pool for rpc messags, monitor hyperviors > > ectra. > > > > if the remote debugger is used it disables patching the thread lib > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L44-L46 > > > > that is prartly why usign the remote debugger is kind fo buggy in itsself. > > the code is not inteded to work that way. > > > > these fucntionlatiy mainly exist to allow debuging tests not the actulaly > > running code. > > > > if i need to debug test i normaly jsut comment out > > https://github.com/openstack/nova/blob/master/nova/monkey_patch.py#L83-L89 > > and run the test in my ides(pycharm,vscodes) test debbuger. > > i normally code in nano/emacs but is do use ides if i really cant figure > > out what happening > > however i genreally start with just adding log lines. > > > > when i first started workign on opentack in 2013 i very much liked using > > an ide and debugger > > and i woudl often spend time trying to get nova to work in a debugger. > > > > over tiem i generally found that that was not worth the effort. > > keepign a devstack deployment working that way was often more work then > > doing print staement > > or better writing functional or unit test to repoduce the problem. > > > > you previous mails ask about debuging nova's api. > > > > if you want to do that its one of the easiest thigns to debug. > > first do _not_ use the wsgi script > > adding apache of modwsgi is just going to make your life harder. > > > > just disable eventlet monkey patching, by commetiing or usign the env > > varible > > and run the nova-api console script directly under pdb or your debugger of > > choice. > > that will allow you to basiclly single step debug without any issues as it > > will not be using eventlet > > anymore and it has not infinitly loops or perodic tasks so it mostly just > > works when not using eventlet. > > > > that becase it didnt use eventlet at all in the api until about 2016 and > > even today it only really uses it in one place > > in my eventlet removal serise i split nova in to the part that use > > eventlet directly and the parts that dont in the > > second path > > > > https://review.opendev.org/c/openstack/nova/+/904424 > > > > anything under nova/cmd/eventlet/ needs eventlet today > > everything under nova/cmd/standalone/ does not. > > > > the approch of doign an api call and following it in a debugger is not a > > good way in my expericne to learn how somethign > > like nova really works. it can hellp to a limtied degree but you cant > > eaislly debug across multipel processes. > > a minium nova deployment has 4 process typically more. > > > > you can try but each openstack service (nova, neutorn, cinder, ...) is > > typically compised of several micocservices > > some like keystone and palcement are just rest apis in front of a db and > > dont use eventlet at all so are trivial to > > debug as a result. the rest have non tirival runtime interactions that > > cant be as eaislly decompsed. > > becuase of that nova and other project invest in our functional tests to > > allwo ues to emulate that multi process env > > and better reason about the code. > > > > the debugging functionality is documetned in the contibutor guide > > > > https://github.com/openstack/nova/blob/master/doc/source/contributor/develo… > > but we dont really adverstise that because none of the core maintainers > > use that any more and it has not been maintained > > for the better part of a decade. since we cant use this type of debugging > > with our ci system > > if we cant debug why a failure happens form the ci logs we add more logs > > because thyat will be useful for future us > > and operators in production. so our focus is on logging/monitoring for > > production rather then for developer eases > > simply because of the large techdebt that eventlet creates in this aready. > > > > i know nuetorn has had some better look with using debuggers then nova has, > > > > keystone/placmnet are eventlest free adn are just python web apps/rest > > apis so should "just work" > > in any python debugger. > > > > nova is prehaps the hardest to debug because of the legacy of the proejct > > so its not the best palce to start looking > > how this works. too be clear i have ran nova api, nova-conductor, nova > > scheduler and nova-compute all in 4 ide windows > > with 4 debuggers at once > > it was proably 3 year of working on openstack before i understood how to > > hack that ot work and since i learned > > how to properly debug without a debuggger, logs, code inspection, > > unit/fucntioal testing > > i have nver really need to do that again. > > > > > > gaining the ablity to run openstack in a debugger simply again, in a > > maintainable way is deffienly one of the reason im > > looking forward to removing eventlet but its still a lot of work. > > im not sure how helpful this was but this is proably as clsoe to a blog > > post as you will get. > > > > > > > > On Thu, 15 Aug 2024, 01:04 Jeremy Stanley, <fungi(a)yuggoth.org> wrote: > > > > > > > On 2024-08-14 22:38:43 +0530 (+0530), engineer2024 wrote: > > > > > How many more releases are you planning to include eventlet in > > > > > them? Seems they are removing any support for further development > > > > > according to their official website. So, is the community thinking > > > > > of any alternatives? > > > > [...] > > > > > > > > Just to expand on the first part of Sean's reply, most of what you > > > > want to know is probably answered here: > > > > > > > > > > https://governance.openstack.org/tc/goals/proposed/remove-eventlet.html > > > > > > > > -- > > > > Jeremy Stanley > > > > > > > >

11 months, 4 weeks

Re: [all][tc] Eventlet migration: Call to action

by Herve Beraud

Le jeu. 16 mai 2024 à 13:31, <smooney(a)redhat.com> a écrit : > On Thu, 2024-05-16 at 10:27 +0200, Herve Beraud wrote: > > Thanks Jay for this thread and for summarizing the T.C feeling. > > My following sentences are not directly addressed to you (Jay), but to > > others T.C members. > > > > Le mar. 14 mai 2024 à 18:28, Jay Faulkner <jay(a)gr-oss.io> a écrit : > > > > > Hi all, > > > > > > As you may have read in Goutham's TC summary, the existing eventlet > > > goal[0] is unlikely to gain consensus to merge. This appears to be due > to > > > the difference in needs between OpenStack projects, and belief that > it's > > > unlikely a single solution would serve them all. > > > > > > > I might agree with the fact it's unlikely a single solution would serve > > them all. I was going to update some sentence to make this optionality > more > > stronger: > > > https://review.opendev.org/c/openstack/governance/+/902585/comment/bb363bda… > > > > Also, I was going to propose some adaptations to introduce futurist and > > executors: > > > https://review.opendev.org/c/openstack/governance/+/902585/comment/6b23377c… > > speaking of executors for nova we have approved a specless bluepirnt to > adress some of the low > hanging fruit > https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 Nice to see this initiative, maybe we could reference it (in the proposal or elsewhere) and give additional details to offer a way to go to others. Again, I'm not married to asyncio. To initiate this topic we had to choose a solution, I chose Asyncio, but any proposed solution is welcomed, as long as it helps to exit from the Eventlet dead end. > > which is implemented here > https://review.opendev.org/q/topic:%22eventlet-removal-part-1%22 > although i will be revising that at least once more before its ready to > merge. > > the last two patches in the series remove eventlet.tpool and replace its > use with > futurist a thread pool executor instead > > for our specific usage > https://review.opendev.org/c/openstack/nova/+/917962/4/nova/storage/rbd_uti… > that's a relatively simple transformation. > > there are one our too outer uses of eventlet.tpool in nova and i will also > adress the usage in the guestfs module > but i will likely no touch its usage in the libvirt driver as we have a > rather > complex code dedicated to how we interact libvirt that need careful > thought to untangle. > > > > > But, rather than speaking of "a single solution", I'd suggest speaking > > about "a solution by default". > > > > Indeed, many comments made in this proposal strongly adopt Nova's > > perspective. > > Nova is a major piece of Openstack, but Openstack is not only Nova. > > By updating that sentence (see the previous links), we would make this > > proposal optional for teams who have the capabilities to design their own > > alternatives. Nova. > > > > Teams like Nova have resources that other teams may not have. > > yes and no. the nova teams capacity is fairly limited these days. > we have between 5-10 active contributors depending on how you look at it. > refining that by the people that understand our use of eventlet and can > review the code > and factor in have the time rewrite the relevnet code while being upstream > consistently > enought to actually make progress that probably means about 3-5 people > could actually do this. > > at lesat 2 of those people are actively working on finisnhing multi cycle > efforts > and the rest including myself are heavily engaged in dowstream work or > work in other projects > that limits our upstream review and development time for at least the next > 3-6 months. > > Nova is trying to be pragmatic that we do not have enough contributor to > rewrite the part of nova that would require it to asyncio without heavily > impacting the projects > viablity. > > so we are looking at using futurist executors instead because its a lot > less work to achieve > the goal of not using eventlet while not prohibiting us form exploring > asyncio or > other options eventually if it makes sense. > > if we had to replace eventlet with asyncio im not confident we could > complete that > for nova in the next 12 -18 months. not without effectively stopping all > other development. > we just don't have the review or code writing capacity to do something > that large and complex > while actually ensuring we don't introduce regressions if we were doing > anything else. > > > > T.C members should consider this difference in resources between teams. > > > > T.C members are elected to adopt a global point of view, and not a team > > based point of view: > > > https://governance.openstack.org/tc/reference/principles.html#openstack-fir… > > > > Else, without any default optional solution, teams with lower resources > > will be deprived of any solutions... > > > > > Teams should not consider the use of their surplus as legitimate when > other > > teams are deprived of what is necessary. > > > > As a community member, this is the kind of mindset that I expect from the > > Openstack governance. > > > > > > > Our current situation is: > > > 1) Eventlet usage must be discontinued. The temporary maintainers are > just > > > that -- temporary -- and we should not expect eventlet to continue to > be a > > > valid path moving forward. > > > 2) We have basic agreement that shared tooling, such as oslo libraries, > > > should not dictate a specific threading model. > > > > > > > If by "should not dictate a specific threading model", you mean that Oslo > > forced the use of async/await, then I STRONGLY disagree about 2)... > > Please show me where, in the proposed goal, oslo libraries dictate a > > specific threading model? => > > https://review.opendev.org/c/openstack/governance/+/902585 > that more a refence ot not forcing async await rather then the internal > impelmations. > i.e. dont require your users to use asynio > > > > FYI the libs migration is detailed in the "How to migrate a library" > > section (around line 646 of the proposed goal). > > And a living example of how the libs would work is available at > > https://github.com/4383/snippets/blob/main/python/facade/facade.py > > that snipit seams to assume its oke to create an asyncio eventlop on the > fly > in any geiven invocation which is proably going to be too slow for many > usecases > > i.e. if oslo log did that every time we logged that would be a problem. > i have not messuered the startup time of the asynio eventloop but i expect > its more expensive > then a trival python function call. > > if that loop was reused between calls to the lib or even kicked out into > its own thread > perhaps with a graceful shutdown if noting is run for an extended period > of time like privsep, > then i can see that variation fo the facade pattern working. > we ould have to mussure the overhad of _iter_coroutine > > > https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_runnin… > does appear to be thread safe so in principal it should not cause any > cross thread interactions. > that was one of my concerns when lookign at the snipit initally. > Another option for libraries, would be to implement in various oslo libraries, new drivers/backends fully based on third parties libs who are based on asyncio, and in parallel to continue to maintain the current existing drivers/backends. Mike Bayer suggested that oslo.db may provide an asyncio interface for applications that want to make use of this. To that end the main working interface in oslo.db is oslo_db.sqlalchemy.enginefacade, so we would propose a layer on top called oslo_db.sqlalchemy.asyncio_enginefacade or similar. it would provide the async versions of SQLAlchemy objects as well as provide for asyncio context managers. The existing interface will remain there in parallel. Oslo.messaging may implement a new rabbitmq driver based on https://github.com/Polyconseil/aioamqp and keep the existing driver around. I agree we should not want to close the door for alternatives like threads and futurist, If futurist avoid us wasting too much time removing Eventlet, then let's go with this kind of option, but in all cases I think we should not close the door to asyncio either. Using Asyncio may make sense in many scenarios. If libraries do not provide async based interfaces in parallel with the sync based interfaces, then, I think we more or less definitely close the door of using Asyncio in Openstack or at list we really limit its introduction. > > > > > To conclude, is it still worth it to spend time updating this proposal > > again or is your decision definitive? > > > > Thanks Sean -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 year, 2 months

openstack-discuss search results for query "#eventlet-removal"