openstack-discuss search results for query "#eventlet-removal"

openstack-discuss@lists.openstack.org

149 messages

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-20)

by Goutham Pacha Ravi

Hello Stackers, This week is milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. We've been hard at work implementing the action items from the recent Virtual Project Teams Gathering (PTG) [2]. This past week, we didn’t pursue any new governance changes, though several remain in draft as follow-ups from the PTG. We’re also excited about the extended deadline for the Call for Proposals for the OpenInfra Days event at SCaLE 2025. The new deadline is Nov 15, 2024 [3]. === Weekly Meeting === Last week, the Technical Committee met concurrently on video [4] and IRC [5]. Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenInfra VMware Migration Working Group [6] joined us to discuss the group's charter and activities so far. The working group’s goal is not only to address gaps and workarounds but also to influence the future development of OpenStack software. The group has published a whitepaper [7] for operators planning to migrate cloud workloads. The TC expressed interest in collaboration between the working group and the current maintainers of the VMware hypervisor driver in OpenStack Nova. We also highlighted low contributor numbers in several critical project teams mentioned in the whitepaper, including those maintaining OpenStack Compute High Availability (Masakari), OpenStack Storage Backup and Restore (Freezer), OpenStack Resource Optimization (Watcher), and OpenStack Database Service (Trove). The working group plans to hold regular meetings soon and will share details via this mailing list. The TC reviewed the maintenance status of the OpenStack documentation styling extension for Python Sphinx, openstackdocstheme. This library is outdated, as it depends on older versions, but there aren’t enough maintainers to review changes, which could impact documentation across all projects. Ideally, maintainers of this library should have expertise in Sphinx and JavaScript, making the TC a less-than-ideal group to support it. We encourage migrating documentation away from this custom tool, as Sphinx styling extensions have advanced and may now suffice without heavy customization. While there’s no immediate migration plan, we’re looking for volunteers to lead this effort. In the meantime, we’ll raise concerns with the Oslo project team about the lack of reviews for openstackdocstheme changes. The next meeting of the OpenStack Technical Committee is today (2024-11-12) on OFTC's #openstack-tc channel. You can find the agenda on the Meeting wiki [8]. I hope you'll be able to join us. === Governance Proposals === ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Superuser compilation of PTG summaries: https://superuser.openinfra.dev/articles/october-2024-openinfra-ptg-summary/ [3] OpenInfra Days CFP: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [4] TC Meeting Video Recording, 2024-11-05: https://youtu.be/o_w3OutGEfM [5] TC Meeting IRC Log 2024-11-05: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-05-18.04.log.html [6] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [7] VMware Migration to OpenStack White Paper: https://www.openstack.org/vmware-migration-to-openstack-white-paper [8] TC Meeting Agenda, 2024-11-12: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

9 months

Re: [watcher] 2025.2 Flamingo PTG summary

by Douglas Viroel

Hi Dmitriy, Glad to hear that you plan to revive Watcher role support in OSA. Right, if you plan to run tempest scenarios tests, then you will need at least an additional compute node. Only Gnocchi and Prometheus datasources have support for metrics injection in watcher-tempest-plugin, which allows us to execute different strategy tests. We have CI jobs running scenario tests for both data sources, if you want to take a look [1][2]. Let us know if you need anything. [1] https://zuul.opendev.org/t/openstack/builds?job_name=watcher-tempest-strate… [2] https://zuul.opendev.org/t/openstack/builds?job_name=watcher-prometheus-int… BR, On Wed, Apr 16, 2025 at 5:04 PM Dmitriy Rabotyagov <noonedeadpunk(a)gmail.com> wrote: > Hey, > > Have a comment on one AI from the list. > > > AI: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone > steps up to maintain them, which should include a minimal CI job running. > > So eventually, on OpenStack-Ansible we were planning to revive the Watcher > role support to the project. > How we usually test deployment, is by spawning an all-in-one environment > with drivers and executing a couple of tempest scenarios to ensure basic > functionality of the service. > > With that, having a native OpenStack telemetry datastore is very > beneficial for such goal, as we already do maintain means for spawning > telemetry stack. While a requirement for Prometheus will be unfortunate for > us at least. > > While I was writing that, I partially realized that testing Watcher on > all-in-one is pretty much impossible as well... > > But at the very least, I can propose looking into adding an OSA job with > Gnocchi as NV to the project, to show the state of the deployment with this > driver. > > On Wed, 16 Apr 2025, 21:53 Douglas Viroel, <viroel(a)gmail.com> wrote: > >> Hello everyone, >> >> Last week's PTG had very interesting topics. Thank you all that joined. >> The Watcher PTG etherpad with all notes is available here: >> https://etherpad.opendev.org/p/apr2025-ptg-watcher >> Here is a summary of the discussions that we had, including the great >> cross-project sessions with Telemetry, Horizon and Nova team: >> >> Tech Debt (chandankumar/sean-k-mooney) >> ================================= >> a) Croniter >> >> - Project is being abandoned as per >> https://pypi.org/project/croniter/#disclaimer >> - Watcher uses croniter to calculate a new schedule time to run an >> audit (continuous). It is also used to validate cron like syntax >> - Agreed: replace croniter with appscheduler's cron methods. >> - *AI*: (chandankumar) Fix in master branch and backport to 2025.1 >> >> b) Support status of Watcher Datasources >> >> - Only Gnocchi and Prometheus have CI job running tempest tests (with >> scenario tests) >> - Monaska is inactive since 2024.1 >> - *AI*: (jgilaber) Mark Monasca and Grafana as deprecated, unless >> someone steps up to maintain them, which should include a minimal CI job >> running. >> - *AI*: (dviroel) Document a support matrix between Strategies and >> Datasources, which ones are production ready or experimental, and testing >> coverage. >> >> c) Eventlet Removal >> >> - Team is going to look at how the eventlet is used in Watcher and >> start a PoC of its removal. >> - Chandan Kumar and dviroel volunteer to help in this effort. >> - Planned for 2026.1 cycle. >> >> Workflow/API Improvements (amoralej) >> ============================== >> a) Actions states >> >> - Currently Actions updates from Pending to Succeeded or Failed, but >> these do not cover some important scenarios >> - If an Action's pre_conditions fails, the action is set to FAILED, >> but for some scenarios, it could be just SKIPPED and continue the workflow. >> - Proposal: New SKIPPED state for action. E.g: In a Nova Migration >> Action, if the instance doesn't exist in the source host, it can be skipped >> instead of fail. >> - Proposal: Users could also manually skip specific actions from an >> action plan. >> - A skip_reason field could also be added to document the reason >> behind the skip: user's request, pre-condition check, etc. >> - *AI*: (amoralej) Create a spec to describe the proposed changes. >> >> b) Meaning of SUCCEEDED state in Action Plan >> >> - Currently means that all actions are triggered, even if all of them >> fail, which can be confusing for users. >> - Docs mention that SUCCEEDED state means that all actions have been >> successfully executed. >> - *AI*: (amoralej) Document the current behavior as a bug (Priority >> High) >> - done: https://bugs.launchpad.net/watcher/+bug/2106407 >> >> Watcher-Dashboard: Priorities to next release (amoralej) >> =========================================== >> a) Add integration/functional tests >> >> - Project is missing integration/functional tests and a CI job >> running against changes in the repo >> - No general conclusion and we will follow up with Horizon team >> - *AI*: (chandankumar/rlandy) sync with Horizon team about testing >> the plugin with horizon. >> - *AI*: (chandankumar/rlandy) devstack job running on new changes for >> watcher-dashboard repo. >> >> b) Add parameters to Audits >> >> - It is missing on the watcher-dashboard side. Without it, it is not >> possible to define some important parameters. >> - Should be addressed by a blueprint >> - Contributors to this feature: chandankumar >> >> Watcher cluster model collector improvement ideas (dviroel) >> ============================================= >> >> - Brainstorm ideas to improve watcher collector process, since we >> still see a lot of issues due to outdated models when running audits >> - Both scheduled model update and event-based updates are enabled in >> CI today >> - It is unknown the current state of event-based updates from Nova >> notification. Code needs to be reviewed and improvements/fixes can be >> proposed >> - e.g: https://bugs.launchpad.net/watcher/+bug/2104220/comments/3 >> - We need to check if we are processing the right notifications of if is a >> bug on Nova >> - Proposal: Refresh the model before running an audit. A rate limit >> should be considered to avoid too many refreshments. >> - *AI*: (dviroel) new spec for cluster model refresh, based on audit >> trigger >> - *AI:* (dviroel) investigate the processing of nova events in Watcher >> >> Watcher and Nova's visible constraints (dviroel) >> ==================================== >> >> - Currently, Watcher can propose solutions that include server >> migrations that violate some Nova constraints like: scheduler_hints, >> server_groups, pinned_az, etc. >> - In Epoxy release, Nova's API was improved to also show >> scheduler_hints and image_properties, allowing external services, like >> watcher, to query and use this information when calculating new solutions. >> - >> https://docs.openstack.org/releasenotes/nova/2025.1.html#new-features >> - Proposal: Extend compute instance model to include new properties, >> which can be retrieved via novaclient. Update strategies to filter invalid >> migration destinations based on these new properties. >> - *AI*: (dviroel) Propose a spec to better document the proposal. No >> API changes are expected here. >> >> Replacement for noisy neighbor policy (jgilaber) >> ==================================== >> >> - The existing noisy neighbor strategy is based on L3 Cache metrics, >> which is not available anymore, since the support for it was dropped from >> the kernel and from Nova. >> - In order to keep this strategy, new metrics need to be considered: >> cpu_steal? io_wait? cache_misses? >> - *AI*: (jgilaber) Mark the strategy as deprecated during this cycle >> - *AI*: (TBD) Identify new metrics to be used >> - *AI*: (TBD) Work on a replacement for the current strategy >> >> >> Host Maintenance strategy new use case (jeno8) >> ===================================== >> >> - New use case for Host Maintenance strategy: instance with ephemeral >> disks should not be migrated at all. >> - Spec proposed: >> https://review.opendev.org/c/openstack/watcher-specs/+/943873 >> - New action to stop instances when both live/cold migration are >> disabled by the user >> - *AI*: (All) Review the spec and continue with discussion there. >> >> Missing Contributor Docs (sean-k-mooney) >> ================================ >> >> - Doc missing: Scope of the project, e.g: >> https://docs.openstack.org/nova/latest/contributor/project-scope.html >> - *AI*: (rlandy) Create a scope of the project doc for Watcher >> - Doc missing: PTL Guide, e.g: >> https://docs.openstack.org/nova/latest/contributor/ptl-guide.html >> - *AI*: (TBD) Create a PTL Guide for Watcher project >> - Document: When to create a spec vs blueprint vs bug >> - *AI*: (TBD) Create a doc section to describe the process based on >> what is being modified in the code. >> >> Retrospective >> ========== >> >> - The DPL approach seems to be working for Watcher >> - New core members added: sean-k-mooney, dviroel, marios and >> chandankumar >> - We plan to add more cores in the next cycle, based on reviews >> and engagement. >> - We plan to remove not active members in the 2 last cycles >> (starting at 2026.1) >> - A new datasource was added: Prometheus >> - Prometheus job now also runs scenario tests, along with Gnocchi. >> - We triaged all old bugs from launchpad >> - Needs improvement: >> - current team is still learning about details in the code, much >> of the historical knowledge was lost with the previous maintainers >> - core team still needs to grow >> - we need to focus on creating stable releases >> >> >> Cross-project session with Horizon team >> =============================== >> >> - Combined session with Telemetry and Horizon team, focused on how to >> provide a tenant and an admin dashboard to visualize metrics. >> - Watcher team presented some ideas of new panels for both admin and >> tenants, and sean-k-mooney raised a discussion about frameworks that can be >> used to implement them >> - Use-cases that were discussed: >> - a) Admin would benefit from a visualization of the >> infrastructure utilization (real usage metrics), so they can identify >> bottlenecks and plan optimization >> - b) A tenant would like to view their workload performance, >> checking real usage of cpu/ram/disk of instances, to proper adjust their >> resources allocation. >> - c) An admin user of watcher service would like to visualize >> metrics generated by watcher strategies like standard deviation of host >> metrics. >> - sean-k-mooney presented an initial PoC on how a Hypervisor Metrics >> dashboard would look like. >> - Proposal for next steps: >> - start a new horizon plugin as an official deliverable of >> telemetry project >> - still unclear which framework to use for building charts >> - dashboard will integrate with Prometheus, as metric store >> - it is expected that only short term metrics will be supported (7 >> days) >> - python-observability-client will be used to query Prometheus >> >> >> Cross-project session with Nova team >> ============================= >> >> - sean-k-mooney led topics on how to evolve Nova to better assist >> other services, like Watcher, to take actions on instances. The team agreed >> on a proposal of using the existing metadata API to annotate instance's >> supported lifecycle operations. This information is very useful to improve >> Watcher's strategy's algorithms. Some example of instance's metadata could >> be: >> - lifecycle:cold-migratable=true|false >> - ha:maintenance-strategy:in_place|power_off|migrate >> - It was discussed that Nova could infer which operations are valid >> or not, based on information like: virt driver, flavor, image properties, >> etc. This feature was initially named 'instance capabilities' and will >> require a spec for further discussions. >> - Another topic of interest, also raised by Sean, was about adding >> new standard traits to resource providers, like PRESSURE_CPU and >> PRESSURE_DISK. These traits can be used to weight hosts when placing new >> VMs. Watcher and the libvirt driver could work on annotating them, but the >> team generally agreed that the libvirt driver is preferred here. >> - More info at Nova PTG etherpad [0] and sean's summary blog [1] >> >> [0] https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d >> [1] https://www.seanmooney.info/blog/2025.2-ptg/#watcher-topics >> >> >> Please let me know if I missed something. >> Thanks! >> >> -- >> Douglas Viroel - dviroel >> > -- Douglas Viroel - dviroel

3 months, 3 weeks

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-15)

by Goutham Pacha Ravi

Hello Stackers, We're 15 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. During the last week, the Technical Committee continued discussing the state of "unmaintained" code branches across OpenStack repositories and election officials proposed dates for the combined TC+PTL elections for the next release cycle [2]. We have decided to cancel the weekly meetings previously scheduled for 24th December and 31st December in lieu of the holidays. === Weekly Meeting === The OpenStack Technical Committee met over IRC on 2024-12-10 [3]. We revisited challenges with using CentOS 10 in the community CI system due to lack of AVX/AVX2 CPU flags on current cloud providers. Discussions revolved around finding providers with suitable hardware or reducing reliance on CentOS testing. We then revisited the state of "unmaintained" branches across OpenStack repositories. The oldest "unmaintained" branch tracks the "victoria" release (GA: 2020-10-14). The consensus amongst TC members was to enforce the policy to default old branches to EOL unless explicitly opted-in with required maintenance assurances. To do this, we need tooling to automate the transition, and a way to track and validate volunteer opt-ins for unmaintained branches. Volunteers stepped up to transition branches up until the 2023.1 ("Antelope") release to "end-of-life", beginning with the Victoria release branch [4]. If you're interested in keeping a particular branch open, for any repository, please express your interest directly on the gerrit change. All EOL transition patches will be merged after 1 month of community review. The next meeting of the OpenStack Technical Committee is on 2024-12-17 at 1800 UTC. This meeting will take place in the #openstack-tc channel on OFTC's IRC server. Please find the meeting agenda on the wiki [5]. Please remember that you can propose topics for upcoming TC meetings by directly editing the wiki. If you're unable to do so, please holler at me (IRC nick: gouthamr) or in the TC's IRC channel (#openstack-tc). === Governance Proposals === ==== Merged ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 === How to Contact the TC === You can reach the TC in several ways: - Email: send an email with the tag [tc] on this mailing list. - IRC: Ping us using the tc-members keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2/"F" elections proposal: https://review.opendev.org/c/openstack/election/+/937408 [3] TC Meeting IRC Log 2024-12-10: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-10-18.01.log.html [4] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [5] TC Meeting Agenda, 2024-12-17: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

7 months, 4 weeks

Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch

by Yatin Karel

Hi Renu and Shivam, On Thu, Jul 31, 2025 at 9:20 AM Shivam Shukla <shivam.shukla3(a)india.nec.com> wrote: > Hi Yatin, > > Hope you're doing well. > > First of all, thank you once again for your support and valuable input on > the Neutron-related issue we were facing in the Tacker base job. Your > suggestion to remove 'logger' from the Q_ML2_PLUGIN_MECHANISM_DRIVERS configuration > resolved the "Internal Server Error" and Neutron startup problems, as > confirmed in the following build: > https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb > > Just as a gentle reminder — when you get a chance, it would be really > helpful if you could share more details about the root cause of this issue. > Specifically, we’d like to understand why the presence of 'logger' causes > the failure, how removing it resolves the problem, and whether any > functional or behavioral changes are expected as a result after its > (logger) removal. > > It's mainly related to eventlet removal[1], there you can see a couple of patches related to dropping "logger" mechanism drivers. [1] https://review.opendev.org/q/topic:%22eventlet-removal%22+project:openstack… > This explanation will help us document the reasoning clearly in the patch > and provide proper justification for the change. > > Looking forward to your insights. > > Thank You. > > Regards, > Shivam > ------------------------------ > *From:* Shivam Shukla <shivam.shukla3(a)india.nec.com> > *Sent:* Sunday, July 27, 2025 10:35 PM > *To:* Yatin Karel <ykarel(a)redhat.com>; Renu Rani <renu.rani(a)india.nec.com> > *Cc:* Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley < > haleyb.dev(a)gmail.com>; OpenStack Discuss < > openstack-discuss(a)lists.openstack.org>; Pankaj Kumar Pandey < > pankaj.pandey(a)india.nec.com> > *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron > in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch > > Hi Yatin, > Thank you for your input regarding the shared issues. > > JFYI... I tried your suggested recommendation i.e., remove "logger" from > Q_ML2_PLUGIN_MECHANISM_DRIVERS in Tacker base job configuration. It > resolves the issues ("Internal Server Error" and "Neutron did not start") > related to Neutron. > > Please refer this build result: > https://zuul.opendev.org/t/openstack/build/48d742463f02457c8a99dd0fed0db1bb > > Also, I would also appreciate it if you could share some details about the > root cause of the issue and explain how removing the 'logger' resolves it." > > Thank you once again for your inputs. > > Regards, > Shivam > ------------------------------ > *From:* Yatin Karel <ykarel(a)redhat.com> > *Sent:* Monday, July 21, 2025 11:13 PM > *To:* Renu Rani <renu.rani(a)india.nec.com> > *Cc:* Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley < > haleyb.dev(a)gmail.com>; OpenStack Discuss < > openstack-discuss(a)lists.openstack.org>; Shivam Shukla < > shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey < > pankaj.pandey(a)india.nec.com> > *Subject:* Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron > in Tacker Zuul Patches – Suspected Regression in Neutron Master Branch > > You don't often get email from ykarel(a)redhat.com. Learn why this is > important <https://aka.ms/LearnAboutSenderIdentification> > *External Message:* Please be cautious when opening links or attachments > in email > > Hi, > > On Mon, Jul 21, 2025 at 8:22 PM Renu Rani <renu.rani(a)india.nec.com> wrote: > > Dear Takashi San, > > Thanks for your support. > > Please fine below further investigation updates from Tacker: > > 1. Kubernetes-Based Job – tacker-ft-v2-k8s > Error: "AttributeError: module 'setuptools.build_meta' has no attribute > 'get_requires_for_build_editable'. Did you mean: > 'get_requires_for_build_sdist'?" > Root Cause: > The issue is caused by the default setuptools version > (59.6.0) included in the base OS image (Ubuntu Jammy). Since > Global_Venv=False is set in local.conf, DevStack does > not use a virtual environment, leading to the use of the > outdated setuptools version. The required attribute is only available from > setuptools version 64 onward. > Fix: > Set Global_Venv=True in local.conf. This enables the > DevStack virtual environment where setuptools version 80 is installed, > resolving the issue by using the correct setuptools version. > > > 2.OpenStack-Based Job – tacker-ft-v1-compliance-sol > Error#1: > Internal Server Error during execution of the command: > "openstack --os-cloud devstack-admin-demo --os-region RegionOne > router add subnet 20b264df-5f19-4599-a8" > Root Cause: > Below devstack commit started causing this issue. > DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 > --> Seems Devstack code update related to neutron. > [Commit sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] > > Error#2: > [ERROR] Neutron did not start > Root Cause: > Below Neutron commit started causing this issue: > Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 > [Commit SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] > > For this I suggested some changes in the test patch > https://review.opendev.org/c/openstack/tacker/+/952892, hopefully that > will clear it. > > Regards > Renu > > -----Original Message----- > From: Renu Rani <renu.rani(a)india.nec.com> > Sent: Thursday, July 17, 2025 2:56 PM > To: Takashi Kajinami <kajinamit(a)oss.nttdata.com>; Brian Haley < > haleyb.dev(a)gmail.com>; OpenStack Discuss < > openstack-discuss(a)lists.openstack.org> > Cc: Shivam Shukla <shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey < > pankaj.pandey(a)india.nec.com> > Subject: RE: [Neutron][Tacker][Issue Investigation] Issue with Neutron in > Tacker Zuul Patches – Suspected Regression in Neutron Master Branch > > Dear Takashi San, > > Thanks a lot for your response and really appreciate sharing your inputs. > > > Just to update there are two types of Tacker jobs is check pipeline: one > is Kubernetes-based, and the other is OpenStack-based. > > [1] Kubernetes-based Job: tacker-ft-v2-k8s > It appears that you reviewed the logs for this job, which is currently > failing due to the following error: > "AttributeError: module 'setuptools.build_meta' has no attribute > 'get_requires_for_build_editable'. Did you mean: > 'get_requires_for_build_sdist'?" > > We understand that this error is not related to Neutron changes, but > rather due to an incompatibility between pip and setuptools, likely caused > by recent packaging updates. We're actively investigating the root cause of > this issue. > > [2] OpenStack-based Job: tacker-ft-v1-compliance-sol > Our original request was regarding this job. It fails due to an > "Internal Server Error" during the execution of the following command: > "openstack --os-cloud devstack-admin-demo --os-region RegionOne router > add subnet 20b264df-5f19-4599-a8" > Zuul Log (controller node): > https://zuul.opendev.org/t/openstack/build/970baccf521846ec83441c1f18898199 > > > To debug further, we tried checking out the stable/2025.1 branch of > DevStack instead of master. However, the controller node now fails with > this error: > "[ERROR] /opt/stack/devstack/functions-common:2366 Neutron did not start" > Zuul Log (controller node): > https://zuul.opendev.org/t/openstack/build/09d39537443640c2a87673590f577829 > > > Identified the specific commits that introduced these issues: > > DevStack Commit (leads to “Internal Server Error”) – May 20, 2025 [Commit > sha: 988ce885d366dd0a7f8a012f9e52ee8f3403ffee] > [ > https://github.com/openstack/devstack/commit/988ce885d366dd0a7f8a012f9e52ee… > ] > > Neutron Commit (leads to “Neutron did not start”) – July 1, 2025 [Commit > SHA: 1971c4a261ca54f077e2dec8b59774602bbbe3f5]] > [ > https://github.com/openstack/neutron/commit/1971c4a261ca54f077e2dec8b597746… > ]] > > > Regards > Renu > > -----Original Message----- > From: Takashi Kajinami <kajinamit(a)oss.nttdata.com> > Sent: Wednesday, July 16, 2025 12:42 PM > To: Brian Haley <haleyb.dev(a)gmail.com>; Renu Rani <renu.rani(a)india.nec.com>; > OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Cc: Shivam Shukla <shivam.shukla3(a)india.nec.com>; Pankaj Kumar Pandey < > pankaj.pandey(a)india.nec.com> > Subject: Re: [Neutron][Tacker][Issue Investigation] Issue with Neutron in > Tacker Zuul Patches – Suspected Regression in Neutron Master Branch > > [You don't often get email from kajinamit(a)oss.nttdata.com. Learn why this > is important at https://aka.ms/LearnAboutSenderIdentification ] > > External Message: Please be cautious when opening links or attachments in > email > > > Looking at the build result[1] in the DNM patch mentioned, I found that > pip failed to install neutron from repo due to the following error. > [1] > https://zuul.opendev.org/t/openstack/build/ef1dbefc5dab40c9a6e13cb4badaa3b2 > > AttributeError: module 'setuptools.build_meta' has no attribute > 'get_requires_for_build_editable'. Did you mean: > 'get_requires_for_build_sdist'? > > I think this indicates the issue is caused by incompatibility between pip > and setuptools which is caused by recent multiple updates in packaging. > > One thing I noticed is that the job still runs in Ubuntu Jammy, although > Jammy was removed from supported operating system in this cycle and should > have been replaced by Ubuntu Noble. Probably you have to first try > migrating these jobs to noble and see if that can resolve the issue. > > On 7/16/25 1:20 AM, Brian Haley wrote: > > Hi Renu, > > > > On 7/15/25 3:16 AM, Renu Rani wrote: > >> Hi Neutron Team, > >> > >> We are currently facing an issue in Tacker Zuul patch jobs build (with > master branch), where the DevStack setup fails during the initial Neutron > network creation step. > >> > >> *Problem statement:* > >> > >> The failure occurs at this point in the stack.sh script, for devstack > environment creation in build process. > >> > >> Below command execution is failing in stack.sh > >> > >> *[openstack --os-cloud devstack-admin-demo --os-region RegionOne > >> router add subnet 20b264df-5f19-4599-a83d-2e76e7cbf5dc > >> d777d04b-93ff-4e94-b6a9-cc8de3e23921]* > > > > I looked quickly at the log files and it looks like you are also using > the networking-sfc code. We just fixed an issue with eventlet there which > could cause a bad interaction with neutron: > > > > https://revi/ > > ew.opendev.org%2Fc%2Fopenstack%2Fnetworking-sfc%2F%2B%2F954633&data=05 > > %7C02%7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7 > > Ccc4532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168427284%7CUnkno > > wn%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa > > W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=olPBCnubJho > > UYWt%2FCi6WoUJA2dzpiHKljXCxWs4lmG8%3D&reserved=0 > > > > I'm not sure if it's related but that change just merged an hour ago, so > it might be worth re-running things to check. > > > > This was the version of networking-sfc I see being built: > > > > Successfully installed networking-sfc-20.1.0.dev3 > > > > And it was pulling-in eventlet (which was just removed in that patch): > > > > Requirement already satisfied: eventlet>=0.27.0 in > > /opt/stack/data/venv/lib/python3.12/site-packages (from > > networking-sfc==20.1.0.dev3) (0.40.1) > > > > If that doesn't help please file a bug against neutron so someone can > look further. > > > > Thanks, > > > > -Brian > > > > > >> > >> ** > >> > >> *Error Message:* > >> > >> /This results in a 500 Internal Server Error, as seen in the logs:/ > >> > >> /[HttpException: 500: Server Error for url: > >> http://10.x/ > >> .x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7cb > >> f5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com% > >> 7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110c > >> %7C0%7C0%7C638882467168459279%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk > >> iOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoy > >> fQ%3D%3D%7C0%7C%7C%7C&sdata=qn%2FaQZX5%2FaGkPyInL8RKDpr1WhliJE%2BpJ90 > >> HANXME%2FQ%3D&reserved=0 > >> <http://0.0.0.10/. > >> x.x.x%2Fnetworking%2Fv2.0%2Frouters%2F20b264df-5f19-4599-a83d-2e76e7c > >> bf5dc%2Fadd_router_interface&data=05%7C02%7Crenu.rani%40india.nec.com > >> %7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4532b813fc4545981297eef82e110 > >> c%7C0%7C0%7C638882467168484477%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG > >> kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo > >> yfQ%3D%3D%7C0%7C%7C%7C&sdata=HhIBD9MAnWFoMCBtXX5FY3k2tfvlKiBmMKDiTFuy > >> iNQ%3D&reserved=0>]/ > >> > >> *Analysis for the issue:* > >> > >> Below error is coming in Neutron API logs: > >> > >> *[ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: cannot notify on > >> un-acquired lock]*** > >> > >> 1. Tried overriding the required-projects in Zuul to use stable/2025.1 > >> branches (instead of master) to identify project(master branch) > >> which is causing root cause. > >> 2. Tacker team has created DNM patch (952892) to invetsigate this in > >> detail. > >> > >> https://rev/ > >> iew.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02% > >> 7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc45 > >> 32b813fc4545981297eef82e110c%7C0%7C0%7C638882467168506073%7CUnknown%7 > >> CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z > >> MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=anYUb6qkQlI%2 > >> Fe1FSOcEtkdsr0iuv3OnNS96Dh2Zl4eY%3D&reserved=0 > >> <https://re/ > >> view.opendev.org%2Fc%2Fopenstack%2Ftacker%2F%2B%2F952892&data=05%7C02 > >> %7Crenu.rani%40india.nec.com%7C5fbac6b252c5459b0e4608ddc438073c%7Ccc4 > >> 532b813fc4545981297eef82e110c%7C0%7C0%7C638882467168521702%7CUnknown% > >> 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 > >> zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0J1axj9cioqQ > >> BuKReXSeCkA8rWpR2kPJ1YWmjQhG4DU%3D&reserved=0> > >> > >> 3. In DNM patch Patchset 11 (on June 29), we have taken devstack > >> stable/2025.1. In stable branch the devstack environment created > >> sucessfully. > >> 4. In DNM patch Patchset 25 (on July 8), with the same devstack > >> override (stable/2025.1), the job now fails earlier during DevStack > >> setup. > >> > >> The logs show that Neutron failed to start, causing the job to fail > across all nodes. > >> > >> As per observations and anlysis seems the recent changes in the Neutron > master branch between June 29 and July 8 might be causing this issue. > >> > >> We’d appreciate your help in reviewing the changes in Neutron's master > branch during that window that could cause this behavior and suggesting any > debugging approaches or patches we can test on tacker. > >> > >> Please let us know if you'd like additional logs or info from the Zuul > job runs. > >> > >> Thanks in advance for your support! > >> > >> Thanks & Regards, > >> > >> Renu Rani(レヌラニ) > >> > >> Mobile: +91-9311888962 > >> > >> E-mail: renu.rani(a)india.nec.com <mailto:renu.rani@india.nec.com> > >> > >> The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > >> It shall not attach any liability on the originator or NEC Corporation > India Private Limited or its affiliates. > >> Any views or opinions presented in this email are solely those of the > author and may not necessarily reflect the opinions of NEC Corporation > India Private Limited or its affiliates. > >> Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of this message without the > prior written consent of the author of this e-mail is strictly prohibited. > >> If you have received this email in error please delete it and notify > the sender immediately. > >> > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > It shall not attach any liability on the originator or NEC Corporation > India Private Limited or its affiliates. > Any views or opinions presented in this email are solely those of the > author and may not necessarily reflect the opinions of NEC Corporation > India Private Limited or its affiliates. > Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of this message without the > prior written consent of the author of this e-mail is strictly prohibited. > If you have received this email in error please delete it and notify the > sender immediately. > > > > -- > Thanks and Regards > Yatin Karel > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > It shall not attach any liability on the originator or NEC Corporation > India Private Limited or its affiliates. > Any views or opinions presented in this email are solely those of the > author and may not necessarily reflect the opinions of NEC Corporation > India Private Limited or its affiliates. > Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of this message without the > prior written consent of the author of this e-mail is strictly prohibited. > If you have received this email in error please delete it and notify the > sender immediately. > -- Thanks and Regards Yatin Karel

1 week, 6 days

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)

by Goutham Pacha Ravi

Hello Stackers, We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. As a contributor/maintainer, you have probably made plans to adjust your goals and project deliverables in light of the likely drop in capacity for the year-end holiday season that's ahead. In the past week, the OpenStack Technical Committee didn't commit any new governance changes; however, we're actively working on our action items from the PTG [2]. === Weekly Meeting === The TC's weekly meeting last week was well attended [3]. Clark Boylan (clarkb) noted that contributors were reaching out to OpenDev Infrastructure Maintainers via IRC and the Zuul email list seeking help to set up and configure "third-party" CI systems to meet the requirements set by several projects (Nova, Neutron, Cinder, Manila, ...). Contributors often lack clear documentation and support for setting up these systems, leading to misplaced queries on Zuul or OpenDev channels. We agreed that there is a lack of centralized, up-to-date documentation regarding CI systems put together to test things outside of the purview of the community's Zuul CI. We have lost mindshare regarding third-party CI setup and maintenance, and this hurts the participation of vendor driver maintainers. The TC would love to see interested parties come together to re-form the "Third-Party CI SIG." If we're unable to meaningfully help contributors with this, we must evaluate whether third-party CI should remain a mandatory requirement for service projects. It was highlighted that Nova maintains a dedicated weekly topic to discuss specific third-party CI issues, a model other projects might emulate. We also discussed kicking off the election planning for the 2025.2 cycle. There will be 5 seats available on the OpenStack TC, and every OpenStack project team will elect a PTL or liaison [4]. Election officials will convene over the next few weeks to propose election dates and other details. There's still time to volunteer to be an election official. Please contact me off-list if you're interested! The next meeting of the OpenStack Technical Committee is today (11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the meeting wiki [5]. I hope you're able to attend. Please remember, you can always propose new topics for this meeting, and it's a great way to synchronously engage with the TC every week. === Governance Proposals === ==== Open for Review ==== rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 ==== Merged ==== Add ubuntu noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] OpenStack TC's Epoxy PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] TC Meeting IRC Log 2024-11-12: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html [4] OpenStack Election website: https://governance.openstack.org/election/ [5] TC Meeting Agenda, 2024-11-19: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

8 months, 3 weeks

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-17)

by Goutham Pacha Ravi

Hello Stackers, This week marks the 2024.2 ("Dalmatian") release trailing deadline [1]. All projects following the cycle-trailing release model [2][3] must release their 2024.2 ("Dalmatian") deliverables by December 5, 2024. No new governance changes were merged in the past week; goal champions and volunteers continued their efforts on the cross-community goals that have been committed to [4]. As Ghanshyam Mann (gmann) highlighted in this mailing list [5], nine project teams need to address CI jobs within their repositories due to the switch of base DevStack (and some tox) jobs to Ubuntu 24.04 LTS. Teams can work around the problem temporarily by pinning the nodeset used to run their failing jobs to Ubuntu 22.04 LTS while they triage and resolve these issues. The Ironic and Neutron teams recently dropped functional testing with the PostgreSQL database. As highlighted at the PTG [6], there is very little motivation within OpenStack's contributor community to support PostgreSQL as a database alternative to MySQL and its derivatives with Oslo.db. The TC has clarified this in a past resolution as well [7]. We are currently identifying places within the documentation where PostgreSQL references need to be removed and aim to address these promptly. If you rely on PostgreSQL, please voice your concerns and consider dedicating time and resources to reviving testing with this database. === Weekly Meeting === TC members met last Tuesday (2024-11-26) on OFTC's #openstack-tc channel [8]. A significant portion of the discussion focused on the state of CI. Members of the Testing and Collaboration Tools SIG highlighted that much of the gate flakiness is due to jobs encountering the new rate limits on DockerHub [9]. Alternatives that may be more suitable for CI usage within open-source communities were discussed. As a result, we encourage project teams to explore these alternatives, as many open-source communities have already done. The OpenStack TC's next weekly meeting is today (2024-12-03). It will be held simultaneously via video (Zoom) and IRC (OFTC's #openstack-tc channel) at 18:00 UTC. The agenda for this meeting, along with participation links, can be found on the wiki [10]. I hope to see you there! === Governance Proposals === ==== Open for Review ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles: https://review.opendev.org/c/openstack/governance/+/935677 - Add ansible-role-httpd repo to OSA-owned projects: https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by the community: https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language: https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal: https://review.opendev.org/c/openstack/governance/+/934936 Thank you for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Cycle-trailing release model: https://releases.openstack.org/reference/release_models.html#cycle-trailing [3] Cycle-trailing projects: https://codesearch.opendev.org/?q=type%3A%20trailing&i=nope&literal=nope&fi… [4] Proposed and selected community goals: https://governance.openstack.org/tc/goals/#community-goals [5] Migrate to noble communication: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [6] OpenStack TC PTG Notes (around line 159): https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b [7] State of community support for databases: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.… [8] TC Meeting IRC Log 2024-11-26: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-26-18.01.log.html [9] DockerHub rate limit changes: https://www.docker.com/blog/november-2024-updated-plans-announcement/ [10] TC Meeting Agenda, 2024-12-03: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

8 months, 1 week

[ironic][ptg] 2025.2 (Flamingo) PTG Summary

by Julia Kreger

Greetings everyone! Last week Ironic had a wildly successful PTG! We averaged around fifteen attendees most days, we drew upwards of 22 attendees for most of Wednesday while we discussed a number of related topics around networking. Overall, networking aspects seem to show the most current interest. This is the result in a change in market ecosystem where vendors are less focused on SDN integrations, and also deliberate limitations inside of ironic in order to limit scope creep and still attempt to meet the majority of infrastructure operator requirements when the original networking multi-tenancy effort was executed back in 2015-2016. Other major topics were Eventlet, some operator feedback on quarks of features, which highlighted some possible bugs and areas for improvement. We also got into some areas which have been a bit contentious in the past, but we reached some reasonable compromises which allowed us to find better paths forward. Monday: We largely discussed where we were at and where we were going. To highlight, ironic-lib has now been retired. Redfish based Graphical Console support merged and we identified some more work was likely needed. Bootable Container support *and* support for artifacts from an OCI container registry support was also viewed as completed during the cycle which ultimately improves options to infrastructure operators who are operating in mixed environments or with mixed requirements outside of the classical "everything is a VM in OpenStack" context. We've improved the linting across the majority of project repositories. Out of band inspection rules likely need more work around ensuring data structures are what we expect, but otherwise we are re-affirming our deprecation of ironic-inspector as a standalone project. Work to support Kea as a DHCP backend has paused for now. The plan exists, but realistically the contributor working in that area has a different focus at the moment which is okay! Container based IPA steps also didn't make it into the Epoxy cycle release, but are almost done already in Flamingo. Schema validation and OpenAPI work is also still underway and we broadly expect this to merge into Flamingo. In-band disk encryption as part of the ``direct`` deploy interface also didn't make any forward progress due to shifting contributor priorities. As a note, the bootable container work did extend this as a possible option, however it remains unclear if that will solve the overall need for the contributor who did propose adding support for encrypted volumes to the ``direct`` interface. Efforts to find an alternative to TinyCore usage in CI also stalled out. Turns out is less of a trivial problem to make a super-low memory ramdisk, and other fixes to the build jobs improved general reliability in the meantime so it is less pressing. We also didn't make any progress on Project Mercury as it was likely framed to try and keep too many options open for operators while we also lost contributor velocity due to some CVE work this past cycle. Having discussed what we achieved, this allowed us to focus on key aspects contributors know they needed to focus on this next cycle. Networking was the biggest topic in this area. Promptly followed by eventlet removal. There was also a consensus that we needed to ensure we're working together a bit better, but also don't explicitly block any one aspect, as there is power in user/operator choice as well. Some possible interest was also expressed around extending network device firmware upgrades, further delineating/improving metrics, possibly supporting more of a "push" firmware update model which more vendors are adopting. We promptly shifted to CI, and discussed challenges and path forward. Generally, there is some interest in trying to make some of our job executions a bit more selective, and also to dial back our reliance upon integration scenario jobs. Overall, it is a broad area of work which will require some further discussion to try and frame an ideal future state. In this we also discussed Centos Stream 10, python versions, and ultimately possible paths forward to at least highlighting other options which may enable maintenance activities to ensure these cases work. Afterwards, we shifted to CI knowledge sharing later in the day outside of the PTG schedule, to help spread overall context among newer members of the team. Tuesday! We started Tuesday with the topic of supporting a use case desired by some operators wishing to "allocate" baremetal nodes into a state which, to Ironic signifies the node has been deployed. The use case behind that is a bit odder, more for research and academic cases where other tools may be desired to meet very specific requirements. The discussion yielded an operator in the scientific space which could benefit, so some ideas and a possible path was identified. Then we got into the world of eventlet and a mini-retrospective of the challenges and identification of future steps. What we thought would be simple, turned out to be the hardest problem. Ironic Python Agent, originally identified as a good candidate for an early eventlet migration, has turned out to be extremely difficult due to it's need to spawn it's own WSGI server late in the process. Upon further reflection; Ironic itself has this issue as well, because we actually have two places a WSGI server might be spun up: ironic-api (obviously) and the conductor when using json-rpc. It was clear that solving that technical problem was the first step in our migration. We found some examples of gunicorn used in this way; that may be what we look to for an initial prototype. We then spoke about node / hardware monitoring patterns that operators are using and interested in. Redfish based hardware supports sending a notification to a web service on a threshold violation so we discussed adding a listener into Ironic that could receive these and integrate this into the current hardware event notifications. We agreed that a spec would be necessary and we would craft one and proceed from there. Further improvements around making it easier for operators to separate monitoring/events of the services from the hardware itself was discussed but oslo_messaging does not allow for this type of split routing today. We spoke about looking further into ironic-prometheus-exporter to see how it could be configured to potentially support this separation. To wrap up Tuesday, we drifted into discussion of the serial console support which Ironic has today. In essence, there is quite a bit of room for possible improvement there and some operators who are not using Nova wouldn't mind this area to see attention moving forward. The broad idea is we might explore creation of a new interface which enables SSH connections to be proxied through to the IPMI Serial-over-LAN capabilities in a BMC. This distinctly would require IPMI, since there really is not a great answer to serial console/interface access with Redfish. Ultimately this may also mean we extend redfish to support an IPMI interface as well, which seems weird, but discussion was in broad agreement that this space has unique challenges where this makes sense. Some initial ideas were entered into an spec document for discussion and refinement as some of the interested parties are also on the perimeter of the Ironic project. Wednesday! Wednesday was our deep dive into networking! Our very first topic was focused on improving the capabilities around the intentional limitations which were encoded in Ironic's early multi-tenant networking work. The early work was mainly focused on ensuring tenant level isolation on to separate L2 domains, such as VLANs. In large part, this boils down to the mapping, scheduling, and ultimately the dynamic assembly of a port grouping (bonded interfaces). We came into this session with a specification document from a discussion which occurred a few weeks prior and there was general consensus in this option and discussion highlighted a few other intertwined aspects which need to be accounted for. We then dove into ACL support for Networking Generic Switch. Overall, consensus resulted. This discussion also highlighted a number of aspects we also likely need to keep in mind, and potentially may start best be described as overall needs for additional documentation. There are also some further discussions which may need to be revisited networking wise. Ultimately, this is very much an "integrated" use case which was only appealing to about half of the attendees, which is typical given the variety of use cases Ironic solves and is able to support. Then we reached the topic of Standalone Switch management. This quickly became a retrospective into why we didn't make progress with Mercury, and then drove into what are the real minimal requirements which would need to be taken into account. Some of the discussion revolved around questions which were semi-answered in the mercury plan itself, to try and provide enough overlap to generally be usable in more than one use model/case while also enabling or preventing fragmentation. This discussion also crossed over into support of DPUs and how to support them, in large part because the overall model is very different. Some discussion also sort of shifted into maybe merging and tools together. Ultimately this topic requires much more discussion, and to make solid progress on networking aspects we need to enable ourselves to move together in two directions at once, while not blocking each other. With this in mind, we're forming up into two sub-teams which will focus on each area with their own advocates/champions. The goal being to report back to the community each week in a quick update style of engagement. Thursday! Thursday marked our final day of PTG for Ironic. We started by wrapping up some of the networking discussions, in large part the focus on how to make progress. We used this time to identify our high level organizational plans, determine broad interest levels and ultimately outreach. One key aspect which was highlighted, is the initial primary focus for some will likely be making progress on eventlet before shifting gears into networking. We then shifted gears to discussing interest and requirements of mapping storage volumes to overall hosts via DPU devices, which somewhat crossed over into the networking topics. The broad idea was presented with a potential need to frame compound drivers around facilitating broadly different configuration actions. For example, directly via SSH to invoke commands, or for example update a CRD in a distinct OpenShift cluster. Overall, the discussion yielded that maybe invoking a CRD update was not worthwhile given the flux those areas can experience along with competing requirements. Ultimately this is an area in early discussion and would require investment in time and hardware to move forward, but there were really no objections to doing so if we could model and extend what already exists in a useful and applicable way. We then drifted back into networking with our next topic, bridging distinctly different types of fabric together. This was raised much more as a question to see if there was an interest/need and/or requirement. In essence, this is similar to the original l2gw project. A huge highlight of embracing or extending into a more ideal model is the reality that if this existed, it might not be necessary for VXLAN to be considered on the physical side. Discussions then shifted to improving servicing interactions. Servicing functionality was added to Ironic during one of the recent past development cycles to enable firmware upgrades to take place on deployed nodes. In discussion, it was highlighted that in the current model this could take five or more reboots to facilitate the deployment as the current mechanism is largely modeled on the use of agent heartbeats. The discussion shifted to how we could avoid this. And the obvious answer was "add a periodic". Then concerns over adding more periodic queries and jobs shifted the discussion into how we can address that challenge. Ultimately, this may end up with us in a better place for tasks which need to follow-up on state changes in hardware and revisit the state before proceeding with the next step to execute. As the final topic, we dove into deploy steps. This really boiled down to "how do I know what will occur" when I ask Ironic to do something along with "how do I know what occurred". Consensus reached that largely the key aspect to highlight back to the user is "what was done" or "what do we expect to be done" in terms of steps. An idea was raised to record this into node history, which is a historical record list of actions/errors to a node which has existed in Ironic for a number of cycles now, but is most useful if you're trying to figure out prior events without going to logs. Consensus was that doing it this way would be a "quick win". At some point, Ironic contributors also sat down and reviewed jammed outside of the PTG schedule to review/discuss and merge existing items sitting awaiting reviews. Overall, the week was extremely productive! Thanks to everyone who took part. Additional thanks to everyone who collaborated on this summary, and everyone who also helped push our project priorities published for this work cycle[3]. Please remember if you have action items to follow-up. And onward to Flamingo! - The Ironic Team [0]: https://etherpad.opendev.org/p/ironic-ptg-april-2025 [1]: https://review.opendev.org/c/openstack/ironic-specs/+/945642 [2]: https://review.opendev.org/c/openstack/ironic-specs/+/946723 [3]: https://specs.openstack.org/openstack/ironic-specs/priorities/2025-2-workit…

3 months, 4 weeks

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)

by Goutham Pacha Ravi

Hello Stackers, Thanks to all moderators and volunteers for compiling virtual PTG notes and sharing summaries through this mailing list. A lot of exciting work seems to be tracked through the action items from these summaries. I hope you were able to review the TC's PTG summary [1]. Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This is typically a bug-targeting milestone for project teams. The following is a summary of this past week's happenings within the OpenStack Technical Committee. === Weekly Meeting === During the past week, the OpenStack Technical Committee met over IRC [3]. We followed up on action items from weeks preceding the Project Teams Gathering. We finalized PTL and DPL appointments for the "mistral" and "watcher" project teams. We spent some time reviewing action items related to translating documentation and user strings within OpenStack services. Automated jobs posting translation updates from Zanata are broken due to incompatibility with newer Python versions. Migration to Weblate remains a priority and the i18n team may require more support to complete this migration. We're collaborating over IRC to identify the needs and next steps. The TC also discussed progress on the community goal to migrate integration testing to Python 3.12 and to the latest LTS version of the operating system used in DevStack test jobs. Like me, you may have wondered why we’re combining these efforts. The reason is that it allows us to have a simpler test matrix despite the number of Python versions and OS distributions that OpenStack software is tested with. While we maintain unit test jobs for each supported Python version, it is impractical to duplicate each integration job for every OS distribution available. Therefore, we choose the latest LTS (e.g., Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of the supported Python versions (Python 3.12). Additionally, each project will have at least one test job for the minimum supported Python version (Python 3.9), so you'll see a test job for the previous LTS release of the OS that will begin in this release and continue running as it transitions to a "stable" release branch. The next meeting of the OpenStack Technical Committee is on 2024-11-05 (today) at 1800 UTC. This is our monthly video meeting too! This meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenStack VMware Migration Working Group [4] will be joining us to discuss the working group's efforts. Meeting information, including other agenda items, is on the meeting wiki [5]. I hope you'll be able to join us. Please remember that you can always propose items to the TC's weekly meeting agenda. === Governance Proposals === ==== Merged ==== Add Watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === How to Contact the TC === You can reach the TC in several ways: 1. **Email:** Send an email with the tag [tc] on this mailing list. 2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. 3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" TC PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] TC Meeting IRC Log 2024-10-29: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html [4] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [5] TC Meeting Agenda, 2024-11-05: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

9 months, 1 week

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-16)

by Goutham Pacha Ravi

Hello Stackers, We're 16 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. OpenStack project teams are planning bug squashes and review jams in the next couple of weeks before contribution activity slows due to end-of-the-year holidays around the world. The OpenStack Technical Committee did not merge any new governance changes this week; however, a few proposals are iterating with community and TC review. === Weekly Meeting === The OpenStack Technical Committee held its monthly video conference last week, which was simultaneously recorded via text on IRC [2][3]. The discussion tracked several TC initiatives for this release cycle [4]. Most notably, the recent migration of base integration jobs to Ubuntu 24.04 LTS and Python 3.12 has caused CI jobs to break for several code repos [5]. I’d like to reiterate that project teams must resolve these issues or temporarily pin the nodeset of their test jobs to "ubuntu-jammy" to buy some time for fixes. We then checked on the status of "unmaintained" branches. The release team wanted the TC to solidify processes for two specific issues: nominating "unmaintained" liaisons and establishing guidelines for retiring these branches. The lack of such processes has resulted in the proliferation of "unmaintained" code branches across the community and failure to implement the TC resolution that created these branches [6]. The oldest unmaintained branch is "unmaintained/victoria", and it’s unclear if anyone benefits from keeping it around. Discussion on cleanup has been delayed due to uncertainty about what to do [7]. Meanwhile, branches like Yoga may still have high usage and active contributors, emphasizing the need for a clear process to decide their fate. The technical committee may consider advising the release team to eliminate branches with low usage, but this doesn’t solve the problem of finding volunteers to maintain high-usage branches. The idea is that projects can opt to maintain more branches if there is sufficient user interest and support, but there is currently no process for this, nor is it clear how to recruit volunteers. We agreed to continue this discussion at the next meeting. However, in the meantime, there were no objections to EOL-ing the Victoria, Wallaby, and Xena branches. Clark Boylan (clarkb) noted an issue with building CentOS Stream 10 images using Disk Image Builder, as they cannot boot on several clouds due to the x86_64 v3 CPU instruction set requirement. Only a few provider clouds, including OpenMetal and Rackspace Flex, meet the CPU requirements to boot CentOS 10. This limitation may affect options for regular test nodes, and project teams are advised to take notice. Until all Zuul/Nodepool cloud providers meet the CPU requirements, teams may need to adjust their testing strategies. Jeremy Stanley (fungi) reminded us that nominations for individual member representatives for the OpenInfra Foundation Board of Directors are still open [8]. Ten nominations from members are required to make it onto the ballot. The OpenStack TC's next weekly meeting is today, 2024-12-10. This will be an IRC meeting hosted on #openstack-tc at 1800 UTC. Please find the agenda on the meeting wiki [9]. I look forward to seeing you there. === Governance Proposals === ==== Open for Review ==== Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting Video Recording, 2024-12-03: https://youtu.be/KlYbFxjhACs [3] TC Meeting IRC Log 2024-12-03: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-03-18.00.log.html [4] TC Tracker for 2025.1: https://etherpad.opendev.org/p/tc-2025.1-tracker [5] Runtime update to Ubuntu 24.04/Python3.12: https://etherpad.opendev.org/p/migrate-to-noble [6] TC Resolution to create "unmaintained" code branches: https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [7] Transition "victoria" code branch to EOL: https://review.opendev.org/c/openstack/releases/+/935373 [8] Nominations for OpenInfra Individual Board Member Elections: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… [9] TC Meeting Agenda, 2024-12-10: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

8 months

Jump to page: