April 2025 - openstack-discuss - lists.openstack.org

[neutron][OVN] Help With Validation Of ML2/OVN
by keerthivassan86＠gmail.com 18 Apr '25

18 Apr '25

Hello Stackers Deployed OVN with Provider network on minimal setup with 3 controllers & 3 compute nodes. We are not using tenant network (only one external network ) for each vlan 2 & Vlan 8 Below is the ping behaviour for vms spread across two hypervisor (different networks ) VM1: 172.168.2.20 (vlan 2 )- SRC-Hyp VM2: 172.168.3.21 (vlan 8 )- Dest-Hyp Source Hypervisor: root@SRC:~# ovs-tcpdump -i patch-br-int-to-provnet-c24e134d-1a3c-4733-941e-0de548325497 icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovsmi688947, link-type EN10MB (Ethernet), snapshot length 262144 bytes 15:41:25.900113 IP 172.168.2.20 > 172.168.3.21: ICMP echo reply, id 18, seq 1, length 64 Dest Hyp: root@DEST:~# ovs-tcpdump -i patch-br-int-to-provnet-d33f01e8-6fe9-4597-998c-f6e339ba6386 icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovsmi853421, link-type EN10MB (Ethernet), snapshot length 262144 bytes 15:41:25.899394 IP 172.168.3.21 > 172.168.2.20: ICMP echo request, id 18, seq 1, length 64 This is first time with OVS, trying to understand below behaviour why we are seeing echo reply on patch-br-int-to-provnet-c24e134d-1a3c-4733-941e-0de548325497 ? Ideally above will help in transfer traffic from br-int to br-ex via path ( as we have physical interface tagged in br-ex ). Is this expected behaviour ? Can some one please help on this ?

1 0

#eventlet-removal [ptg] 2025.2 Flamingo PTG summary
by Herve Beraud 18 Apr '25

18 Apr '25

Following the intensive discussions at the April 2025 Project Teams Gathering about the Eventlet removal initiative, this thread summarizes the current status, challenges, and strategies of all the OpenStack teams with a transversal perspective. A full and detailed report is available at: https://removal.eventlet.org/guide/openstack/flamingo-ptg/ ## Python 3.13 Compatibility The countdown has begun! As documented in the April 2025 PTG discussions, the OpenStack community is accelerating efforts to remove Eventlet dependencies across all projects. This initiative has become critical due to Eventlet's compatibility problems with Python 3.13 and the upcoming "GILectomy" (PEP 703). With Ubuntu 2025.4 planning to ship Python 3.13 by default, we're facing a hard deadline for this work https://discourse.ubuntu.com/t/plucky-puffin-release-schedule/36461 ## Migration Status ### Significant Progress - Octavia: Fully migrated since 2017! Their approach is now documented as a community case study. - Mistral: Almost there with a comprehensive migration approach. - Neutron: Significant progress with numerous patches already merged. - Glance: Can be deployed without Eventlet, though some optional features still need work. ### Work in Progress - Nova: Planning a service-by-service migration with dual-mode support during transition. - Swift: Evaluating alternatives with impressive performance results (FastWsgi showing 10x better performance!). - Manila, Cinder, Heat: All have active plans for the Flamingo cycle. - Designate, Blazar, Watcher: Starting their migration journeys. - Ironic: Facing complex challenges, particularly with the Ironic Python Agent component. ## Technical Spotlight: Oslo.service's Threading Backend A key development is the new threading backend for oslo.service that eliminates the Eventlet dependency: - Review in progress: https://review.opendev.org/c/openstack/oslo.service/+/945720 - No longer provides WSGI support - Each service will need to deprecate implementations dependent on Eventlet's WSGI server ## Migration Strategies: One Size Doesn't Fit All Projects are adopting diverse approaches: - Dual-mode support: Nova and Glance are supporting both Eventlet and native threads during transition - Canary approach: Swift is considering starting with proxy nodes - Component-by-component: Cinder is starting with the Volume Manager - Complete replacement: Heat is planning a full discontinuation of WSGI server implementations ## Nova's Detailed Roadmap Nova has a comprehensive two-cycle plan: Flamingo Cycle (2025.2): - API modernization - Architecture updates with environment variable controls - Performance improvements - Test transition to native threading Guppy Cycle (2026.1): - Core event loop conversion - Adoption of oslo.service's new threading backend ## Issues to Watch: RabbitMQ Heartbeat Problems The problem with RabbitMQ heartbeats story: - Timeouts and API failures in "green" environments - Partial solution using pthreads exists but has logging issues - Track the fix: https://review.opendev.org/c/openstack/oslo.log/+/937729 ## How to Get Involved - Join the #openstack-eventlet-removal channel on OFTC - Review the official guide: https://removal.eventlet.org/ - Look for patches under the "eventlet-removal" topic: https://review.opendev.org/q/prefixtopic:%22eventlet-removal%22 ## Recommended Reading For teams starting their migration: 1. Official goal documentation: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html 2. Migration preparation guide: https://removal.eventlet.org/guide/preparing-for-migration/ 3. Octavia case study: https://removal.eventlet.org/guide/case-studies/octavia/ ## Stay Updated - The complete version of this report is available online at: https://removal.eventlet.org/guide/openstack/flamingo-ptg/ - Track the progress of the migration across all projects at: https://removal.eventlet.org/guide/openstack/#migration-status Thanks for reading! PS: This PTG summary is based on the etherpads following the April 2025 PTG discussions (https://ptg.opendev.org/etherpads.html) For more details, refer to the full report ( https://removal.eventlet.org/guide/openstack/flamingo-ptg/)

3 4

[tacker][ptg]2025.2 Flamingo PTG summary
by Yasufumi Ogawa 18 Apr '25

18 Apr '25

Hi everyone, Thank you for joining to the vPTG sessions last week. We had great discussion for several topics about Tacker's next development items. Here is a summary of the discussion. ## Revise FTs We've revised functional tests for the recent cycles since maintainance costs or running time are getting large. It's a recap of the previous vPTG discussion and we agree to focus on reducing the number of tests by dropping useless ones and setup configurations. ## Enhancing Tacker Testing Coverage in the OpenStack Helm Check Pipeline A proposal for enhancing Tacker testing within the OpenStack Helm (OSH) pipeline by adding more test cases. It will use the OpenStack CLI for Tacker testing, ensuring that no modifications to the OSH test code will be needed even if Tacker's functional tests change. It's also a recap from the previous cycle and we agreed to continue the development. ## Balancing Test Execution Efficiency and Resource Consumption in Tacker Testing It's a sub task of revising functional tests. Since some of usecases in tacker are little bit complicated and several nodes are used for test such a cases. As the result, the resource consumption is higher than we expected. This proposal aims to analyze this issue and propose possible optimizations to balance resource usage and test execution efficiency. ## Tacker Strategy and Guidelines to Support Newer TST Releases and V2 API We're planning to introduce ETSI NFV standard tests called as TST for ensuring Tacker's capabilities are exactly compliant with the standards, but there is a painpoint about supporting versions. The purpose of this proposal is to outline the strategy and guidelines for updating Tacker to support newer TST releases and its v2 API so that we can define how Tacker can collaborate with the TST team to implement support for newer releases. ## Potential new usecase from vRAN feedback Discussion for a new vRAN usecase other than basic ones already implemented in Tacker as a part of cross community collaboration items between O-RAN SC and OpenStack. Tacker is expected to be SMO which is a frontend of O-Cloud infrastructure defined in the O-RAN SC reference architecture model. In a highliy distributed deployment usecase called as D-RAN (Distributed RAN), there is a concern about resource consumption because of limited num of servers is available on each edge site. K8s may not support a case of far controll node on retional site and worker nodes on edge sites. We discussed two key points and will continue the discussion: (1) Tacker manages container via Docer directly by Infra-driver? and (2) How can Tacker ensure to manage container with delay? ## PaaS Manager It's a emerging requirement for Tacker. In ETSI NFV standards, PaaS and its manager are expected as key components for containers. ETSI NFV has specified PSM (PaaS Service Management) to manage LCM of PaaS in clause 10 of ETSI GS NFV-IFA 049. However, there might be several missing point in term of implementation, such as how manages dependency between VNF and PaaS, e.g. order of start/stop, capacity of PaaS, recovery when PaaS is failure. It's also required to provide a parser for PaaS Service Descriptor (PSD). In the vPTG discussion, we agree to continue the discussion to make it clear the actual requirements for managing PSM or describing PSD, and contribute ETSI NFV specifications. ## Add option to enable/disable Cinder volume recreation in v1 Heal request A proposal for enhancing Tacker v1 APIs by enabling to cancel cincer volume is recreated while heal operation. This feature was already implemented in v2, but it's also required to implement in v1 for keeping compatibility. ## Add default_secret_key Option to [vim_keys] for Multi-Master Tacker Deployment Currently, all Tacker nodes need to share or sync the VIM keys across all nodes. Since VIM keys are randomly generated and stored only on the conductor where the VIM registration process occurs, it doesn't work if instantiate or terminate requests are performed from a different conductor, the VIM key is also required on that host. This proposal is for allowing the Tacker administrator to use a pre-generated common key eliminates the need for key sharing or syncing, making the setup simpler. We agreed to the implementation strategy and expect to merge the feature in Flamimngo. Yasufumi

1 0

openstack ansible nova
by engineer2024 18 Apr '25

18 Apr '25

When running the n os-nova playbook, I want to disable need_online_data_migrations in the nova role. I tried this pasing to the playbook '-e need_online_data_migrations=False', but it did not work. tried deleting the fact file for the nova containers at /etc/openstack_deploy/ansible_facts. folder. still it did not work. It is resetting each time to 'True' Finally I hardcoded in the file /etc/ansible/roles/os_nova/tasks/nova_install.yml ---- - name: Install the python venv import_role: name: "python_venv_build" vars: venv_python_executable: "{{ nova_venv_python_executable }}" venv_build_constraints: "{{ nova_git_constraints }}" venv_build_distro_package_list: "{{ nova_devel_distro_packages }}" venv_install_destination_path: "{{ nova_bin | dirname }}" venv_pip_install_args: "{{ nova_pip_install_args }}" venv_packages_to_symlink: >- {{ (nova_services['nova-compute']['group'] in group_names and nova_virt_type != 'ironic') | ternary(nova_compute_kvm_packages_to_symlink, []) }} venv_pip_packages: "{{ nova_venv_packages }}" venv_facts_when_changed: - section: "nova" option: "need_service_restart" value: "True" - section: "nova" option: "need_online_data_migrations" value: "False" - section: "nova" option: "venv_tag" value: "{{ nova_venv_tag }}" when: nova_install_method == 'source' - name: Record local facts for distro path when: nova_install_method == 'distro' block: - name: Record the osa version deployed ini_file: dest: "/etc/ansible/facts.d/openstack_ansible.fact" section: nova option: venv_tag value: "{{ nova_venv_tag }}" mode: "0644" - name: Initialise the upgrade facts ini_file: dest: "/etc/ansible/facts.d/openstack_ansible.fact" section: nova option: "{{ item.name }}" value: "{{ item.state }}" mode: "0644" with_items: - name: "need_service_restart" state: "True" - name: "need_online_data_migrations" state: "False" when: - (install_packages is changed) or (ansible_local is not defined) or ('openstack_ansible' not in ansible_local) or ('nova' not in ansible_local['openstack_ansible']) or ('need_online_data_migrations' not in ansible_local['openstack_ansible']['nova']) or ('need_service_restart' not in ansible_local['openstack_ansible']['nova']) ---------- how to override it without hardcoding it ?

3 5

[neutron] Drivers meeting canceled Friday, April 18th
by Brian Haley 17 Apr '25

17 Apr '25

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as it does not look like we will have quorum, please add things to the wiki to discuss next week. Have a nice weekend! -Brian

1 0

[tc][ptg] OpenStack Technical Committee 2025.2 Flamingo PTG summary
by Goutham Pacha Ravi 17 Apr '25

17 Apr '25

Hello Stackers, Here’s a summary of the OpenStack Technical Committee’s discussions at the 2025.2 “Flamingo” Virtual PTG held April 7–11, 2025. Find detailed notes in our discussion etherpad [1] and recordings on YouTube [2]. Feedback is welcome on this mailing list or in OFTC’s `#openstack-tc` IRC channel, perhaps during our weekly meetings [3]. == Topic: TC Retrospective == Discussion: - We reviewed the TC’s effectiveness in addressing issues raised during the cycle, including leadership gaps, licensing concerns, project guide updates, and communication practices. - Fewer teams were leaderless following the 2025.2 election cycle—thanks to better outreach and clearer election guidance. - There was consensus that the TC guideline discouraging bare “recheck” comments and the need to address persistent gate issues is now better understood. - Licensing FAQs [4] appear outdated. We noted that the TC can escalate licensing questions to the OpenInfra Foundation’s Legal team and should monitor the `legal-discuss` list more actively. - Teams encouraging non-core contributors to become PTLs was praised as a way to grow the contributor base through mentorship. Action Items: - Update the Project Team Guide, removing obsolete sections (e.g., sprint design/process). - Update licensing references across the Project Team Guide and TC governance site. - Encourage public communication, regular meetings, and the use of team-specific IRC or Matrix channels. == Topic: Goals Retrospective == Discussion: - We reviewed the status of ongoing community-wide goals, including progress, blockers, and next steps. - Migrate CI Jobs to Ubuntu Noble [5] - The goal is complete. Teams should now remove OS version pins in their repositories. - Eventlet Migration [6] - Refer to Hervé’s PTG summary [7] for updates. - Privsep Migration [8] - Progress is slow; some minor activity exists in Gerrit, but tracking doesn’t reflect it. - The Oslo team will de-prioritize removing eventlet from `oslo-rootwrap`, as it’s expected to be retired. - Clear guidance on secure, performant `privsep` usage is needed. - FIPS Compliance [9] - CentOS Stream 10 requires x86_64-v3, unsupported by our nodepool providers. - This hardware/runtime mismatch complicates long-term testing with CS9 or Ubuntu LTS. - Alternatives discussed: Debian, Rocky Linux (but Rocky may inherit CS10's hardware constraints). - Red Hat is evaluating a `paramiko` replacement this cycle. - Consistent and Secure RBAC [10] - Progress tracked [here](https://etherpad.opendev.org/p/rbac-goal-tracking#L14). - Phase 1 (personas + drop system scope): 16 projects complete. - Phase 2 (service role): 3 projects. - Phase 3 (manager role): 3 projects. - Ghanshyam Mann (gmann) continues to chair bi-weekly IRC meetings. Action Items: - Rodolfo Alonso Hernandez (ralonsoh) to send a status update on `privsep` and flag possible `rootwrap` deprecation. - Ade lee (ade_lee) to update FIPS goal status. - SRBAC liaisons must attend bi-weekly meetings and track project progress. - Publish guidance on secure, performant `privsep` usage. == Topic: Revisiting “Unmaintained” Branches == Discussion: - The TC discussed the burden and ambiguity around maintaining aging branches. - Project teams often lack resources or interest to maintain older branches; the Release team has pursued mass EOL as a workaround. - We agreed to reduce scope and prioritize core goals (e.g., eventlet removal, CI stability) rather than preserve old branches without active interest. - TC supports giving `openstack-unmaintained-core` full responsibility for EOL decisions. We may seek volunteers earlier in the branch lifecycle. - Broken jobs in unmaintained branches flood Zuul; fixing in-repo configs may not be viable. OpenDev may opt to ignore failing jobs externally. - There’s an unregistered, unlogged `#openstack-unmaintained` IRC channel on OFTC. Action Items: - Document responsibilities of unmaintained branch liaisons and `openstack-unmaintained-core`. - Propose a new process: `openstack-unmaintained-core` sets deadlines per cycle to solicit maintainers for aging branches. If no one steps up, proceed to EOL. == Topic: TC and Community Leaders Forum == Discussion: - Governance proposal under review to expand the scope of the VMT to cover all OpenStack project teams [11]. - The TC must ensure teams prioritize security. Poor handling of vulnerabilities harms community trust and ethical responsibility. - Ideas were proposed to support the VMT, including a pool of core security contacts from contributing organizations. - Projects were reminded to review community goals and prioritize them. Adding goal timelines and cross-project prioritization was suggested. - Teams should use Stackalytics and Bitergia dashboards to track activity and identify areas needing improvement (e.g., review velocity, core participation). Action Items: - Teams must audit their Launchpad “Bug Sharing” settings and confirm their `coresec` groups contain responsible, active contributors. - PTLs and Security Liaisons should ensure they can manage the `coresec` group or delegate access. - Audit deliverables against [Security Process Requirements][12]. - Benchmark project health using Stackalytics/Bitergia. Reinforce OpenStack code review best practices [13][14]. == Topic: Finding Contributors in Some Projects is Hard == Discussion: - QA, Requirements, and Release Management teams remain understaffed. - Despite upstream investment opportunities for 2025 [15], contributor interest is low. - This is a “Tragedy of the Commons” issue: shared work without clear ownership is often deprioritized. - PTG scheduling didn’t support sufficient cross-project collaboration. - The TC was encouraged to reset perfectionist expectations—approve good-enough changes, and avoid expanding review scope unnecessarily. Action Items: - Explore QA Liaisons: trusted contributors mentored into maintaining QA repos. - Avoid nitpicking or scope creep during code reviews. - Promote and participate in the contributor survey [16]. - Document expectations and best practices in the Project Team Guide. == Topic: Splintered Conversations & Communication Barriers == Discussion: - Operators report more engagement on Reddit, Kubernetes Slack, Discord, etc., than on `openstack-discuss` or OFTC IRC. - Some organizations block Matrix due to regulatory compliance, requiring platforms with retention policies. - The OpenInfra Board will not mandate platforms—projects must choose what suits them. - IRC onboarding is hard for newcomers. - Zuul contributors reported smooth onboarding to Matrix. Its e2e encryption may make it more acceptable for compliance-sensitive orgs. - The conversation continued in the `os-operators` PTG, where there was interest in trying Matrix and Meetpad for better engagement. Action Items: - Explore use of a hosted Matrix homeserver by OpenDev as a short-term workaround for those unable to use IRC. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://etherpad.opendev.org/p/r.51cb4e74dd9980932d32e9a9f4e8a0ce (OS TC PTG Etherpad) [2] https://www.youtube.com/playlist?list=PLhwOhbQKWT7XYzqrx21j1uizxNxnbDf5Q (OS TC PTG Recordings) [3] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee (OS TC Weekly Meetings) [4] https://wiki.openstack.org/wiki/LegalIssuesFAQ (OpenStack Licensing FAQs) [5] https://governance.openstack.org/tc/goals/completed/2025.1/migrate-ci-jobs-… (Migrate to Noble goal) [6] https://governance.openstack.org/tc/goals/selected/remove-eventlet.html (Eventlet removal goal) [7] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… (Eventlet PTG Summary) [8] https://governance.openstack.org/tc/goals/selected/migrate-to-privsep.html (Privsep migration goal) [9] https://governance.openstack.org/tc/goals/selected/fips.html (FIPS compliance goal) [10] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… (SRBAC goal) [11] https://review.opendev.org/c/openstack/governance/+/944817 (TC VMT Resolution) [12] https://security.openstack.org/repos-overseen.html#requirements (Security requirements for OpenStack Projects) [13] https://docs.openstack.org/project-team-guide/review-the-openstack-way.html (Code Reviews in the OpenStack Way) [14] https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatt… (Code Review Antipatterns) [15] https://governance.openstack.org/tc//reference/upstream-investment-opportun… (OpenStack Investment Opportunities) [16] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… (Bridging the Gap)

1 0

[cinder][PTG] 2025.2 Flamingo PTG summary
by Jon Bernard 17 Apr '25

17 Apr '25

Hello everyone, thanks for a very productive PTG for the Flamingo cycle. Below is a brief summary of the highlights. The raw etherpad notes are here [1] and both the summary and transcribed etherpad notes are captured in out wiki here [2]. # Eventlet Removal This cycle we will try complete the removal of Eventlet from our codebase. We dicussed our current usage, including calls to sleep() and tpool.Proxy(). We agreed to split the work between Eric and me. I hope we can get these patches up and reviewed with time to spare before the release. An idea to revive our fake volume driver was raised to potentially gain greater confidence in high concurrency test configurations. # Testing Volume Replication Although volume replication is support by a few of our in-tree drivers, there are no integration tests that verify correct behaviour in tempest. Discussions on adding tempest tests lead to Liron being interested in driving this effort. Changes will need to be made to devstack-ceph plugin to support deploying multiple ceph clusters, as well as gate configuration to run all of this in a multi-node environment. There are a few important moving pieces which may take some time to sort out, but we agreed to commit to working towards a solution in this cycle. # Follow-up from Epoxy We have a few patch series that we didn't have time to review by the deadline of the Epoxy cycle. Volume type metadata, encryption format, and dm-clone just to name a few. Effort in reviewing and gathering feedback needs to be prioritized in the first half of the cycle to make sure we can confidently land these in Flamingo. # CI status We should review our 3rd party CI instances to make sure they're actually running, running the correct tests in the context of the related change, and reporting results that can be viewed by a gerrit reviewer. It was noted that some of them fall short of some or all of these requirements. It would be helpful to gather a current status so that we can help vendors with awareness. # 2023.2 Final Stable Release On April 30, the stable/2023.2 branch will have it's final release. This will mark the end of cinderlib. We still have a few backports that need reviews currently, see etherpad for details. # Backup Improvements crohmann raised 2 patches that both improve the speed of cinder-backup's startup routine, and decouple a volume's backup status from it's overall status. Eric and Brian agreed to review to help move these patches forward. # RBD Geometry Option We've have an old patch to allow RBD's disk geometry to be configurable. This reportedly improves performance of Windows guests significantly. We agreed to make sure this gets reviewed in this cycle. # Stephen's Patches We have 3 patches carried over from last cycle that need review. These include an OpenAPI series, a patch to map block storage AZs to Compute AZs, and updates to OSC. It would be really nice to get these reviewed in this cycle. # Tobias' Patches We have several patches from Tobias all related to cinder-backup improvements. Including adding support for backup service to os-availability-zone, support for filters in query parameters, the addition of a summary API, and potentially adding a `type` field to the API to open the posibility of support backups types beyond just `full` or `incremental`. [1] https://etherpad.opendev.org/p/apr2025-ptg-cinder [2] https://wiki.openstack.org/wiki/CinderFlamingoPTGSummary -- Jon

1 0

OpenInfra Summit Europe registration now open
by Jeremy Stanley 17 Apr '25

17 Apr '25

As most of you already know, the OpenInfra Summit Europe this year is October 17-19 at the École Polytechnique Paris-Saclay in France. Registration is now open here: https://summit2025.openinfra.org/ Don't forget, the CFP closes on June 13! If you're planning to submit a talk, there's still time but don't wait too long. For additional info, see this more detailed announcement on the foundation mailing list: https://lists.openinfra.org/archives/list/foundation@lists.openinfra.org/me… -- Jeremy Stanley

1 0

[dev][infra][tact-sig][tc] Image availability in OpenDev
by Jeremy Stanley 17 Apr '25

17 Apr '25

I want to call everyone's attention to an announcement from the OpenDev Collaboratory sysadmins: https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… The quick summary is that help is needed with ongoing maintenance of the following test platform images, or else they may be removed in the not too distant future: * CentOS Stream (x86 and Arm) * Rocky Linux (x86 and Arm) * Ubuntu (Arm only) * Debian (Arm only) In addition, these images have been unmaintained for a long time and can't currently be rebuilt, so without immediate intervention will be removed very soon: * openEuler * Gentoo See the above service-discuss mailing list thread for more details, and follow up there if you can help out. Thanks in advance! -- Jeremy Stanley

1 0

[watcher] 2025.2 Flamingo PTG summary
by Douglas Viroel 17 Apr '25

17 Apr '25

Hello everyone, Last week's PTG had very interesting topics. Thank you all that joined. The Watcher PTG etherpad with all notes is available here: https://etherpad.opendev.org/p/apr2025-ptg-watcher Here is a summary of the discussions that we had, including the great cross-project sessions with Telemetry, Horizon and Nova team: Tech Debt (chandankumar/sean-k-mooney) ================================= a) Croniter - Project is being abandoned as per https://pypi.org/project/croniter/#disclaimer - Watcher uses croniter to calculate a new schedule time to run an audit (continuous). It is also used to validate cron like syntax - Agreed: replace croniter with appscheduler's cron methods. - *AI*: (chandankumar) Fix in master branch and backport to 2025.1 b) Support status of Watcher Datasources - Only Gnocchi and Prometheus have CI job running tempest tests (with scenario tests) - Monaska is inactive since 2024.1 - *AI*: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone steps up to maintain them, which should include a minimal CI job running. - *AI*: (dviroel) Document a support matrix between Strategies and Datasources, which ones are production ready or experimental, and testing coverage. c) Eventlet Removal - Team is going to look at how the eventlet is used in Watcher and start a PoC of its removal. - Chandan Kumar and dviroel volunteer to help in this effort. - Planned for 2026.1 cycle. Workflow/API Improvements (amoralej) ============================== a) Actions states - Currently Actions updates from Pending to Succeeded or Failed, but these do not cover some important scenarios - If an Action's pre_conditions fails, the action is set to FAILED, but for some scenarios, it could be just SKIPPED and continue the workflow. - Proposal: New SKIPPED state for action. E.g: In a Nova Migration Action, if the instance doesn't exist in the source host, it can be skipped instead of fail. - Proposal: Users could also manually skip specific actions from an action plan. - A skip_reason field could also be added to document the reason behind the skip: user's request, pre-condition check, etc. - *AI*: (amoralej) Create a spec to describe the proposed changes. b) Meaning of SUCCEEDED state in Action Plan - Currently means that all actions are triggered, even if all of them fail, which can be confusing for users. - Docs mention that SUCCEEDED state means that all actions have been successfully executed. - *AI*: (amoralej) Document the current behavior as a bug (Priority High) - done: https://bugs.launchpad.net/watcher/+bug/2106407 Watcher-Dashboard: Priorities to next release (amoralej) =========================================== a) Add integration/functional tests - Project is missing integration/functional tests and a CI job running against changes in the repo - No general conclusion and we will follow up with Horizon team - *AI*: (chandankumar/rlandy) sync with Horizon team about testing the plugin with horizon. - *AI*: (chandankumar/rlandy) devstack job running on new changes for watcher-dashboard repo. b) Add parameters to Audits - It is missing on the watcher-dashboard side. Without it, it is not possible to define some important parameters. - Should be addressed by a blueprint - Contributors to this feature: chandankumar Watcher cluster model collector improvement ideas (dviroel) ============================================= - Brainstorm ideas to improve watcher collector process, since we still see a lot of issues due to outdated models when running audits - Both scheduled model update and event-based updates are enabled in CI today - It is unknown the current state of event-based updates from Nova notification. Code needs to be reviewed and improvements/fixes can be proposed - e.g: https://bugs.launchpad.net/watcher/+bug/2104220/comments/3 - We need to check if we are processing the right notifications of if is a bug on Nova - Proposal: Refresh the model before running an audit. A rate limit should be considered to avoid too many refreshments. - *AI*: (dviroel) new spec for cluster model refresh, based on audit trigger - *AI:* (dviroel) investigate the processing of nova events in Watcher Watcher and Nova's visible constraints (dviroel) ==================================== - Currently, Watcher can propose solutions that include server migrations that violate some Nova constraints like: scheduler_hints, server_groups, pinned_az, etc. - In Epoxy release, Nova's API was improved to also show scheduler_hints and image_properties, allowing external services, like watcher, to query and use this information when calculating new solutions. - https://docs.openstack.org/releasenotes/nova/2025.1.html#new-features - Proposal: Extend compute instance model to include new properties, which can be retrieved via novaclient. Update strategies to filter invalid migration destinations based on these new properties. - *AI*: (dviroel) Propose a spec to better document the proposal. No API changes are expected here. Replacement for noisy neighbor policy (jgilaber) ==================================== - The existing noisy neighbor strategy is based on L3 Cache metrics, which is not available anymore, since the support for it was dropped from the kernel and from Nova. - In order to keep this strategy, new metrics need to be considered: cpu_steal? io_wait? cache_misses? - *AI*: (jgilaber) Mark the strategy as deprecated during this cycle - *AI*: (TBD) Identify new metrics to be used - *AI*: (TBD) Work on a replacement for the current strategy Host Maintenance strategy new use case (jeno8) ===================================== - New use case for Host Maintenance strategy: instance with ephemeral disks should not be migrated at all. - Spec proposed: https://review.opendev.org/c/openstack/watcher-specs/+/943873 - New action to stop instances when both live/cold migration are disabled by the user - *AI*: (All) Review the spec and continue with discussion there. Missing Contributor Docs (sean-k-mooney) ================================ - Doc missing: Scope of the project, e.g: https://docs.openstack.org/nova/latest/contributor/project-scope.html - *AI*: (rlandy) Create a scope of the project doc for Watcher - Doc missing: PTL Guide, e.g: https://docs.openstack.org/nova/latest/contributor/ptl-guide.html - *AI*: (TBD) Create a PTL Guide for Watcher project - Document: When to create a spec vs blueprint vs bug - *AI*: (TBD) Create a doc section to describe the process based on what is being modified in the code. Retrospective ========== - The DPL approach seems to be working for Watcher - New core members added: sean-k-mooney, dviroel, marios and chandankumar - We plan to add more cores in the next cycle, based on reviews and engagement. - We plan to remove not active members in the 2 last cycles (starting at 2026.1) - A new datasource was added: Prometheus - Prometheus job now also runs scenario tests, along with Gnocchi. - We triaged all old bugs from launchpad - Needs improvement: - current team is still learning about details in the code, much of the historical knowledge was lost with the previous maintainers - core team still needs to grow - we need to focus on creating stable releases Cross-project session with Horizon team =============================== - Combined session with Telemetry and Horizon team, focused on how to provide a tenant and an admin dashboard to visualize metrics. - Watcher team presented some ideas of new panels for both admin and tenants, and sean-k-mooney raised a discussion about frameworks that can be used to implement them - Use-cases that were discussed: - a) Admin would benefit from a visualization of the infrastructure utilization (real usage metrics), so they can identify bottlenecks and plan optimization - b) A tenant would like to view their workload performance, checking real usage of cpu/ram/disk of instances, to proper adjust their resources allocation. - c) An admin user of watcher service would like to visualize metrics generated by watcher strategies like standard deviation of host metrics. - sean-k-mooney presented an initial PoC on how a Hypervisor Metrics dashboard would look like. - Proposal for next steps: - start a new horizon plugin as an official deliverable of telemetry project - still unclear which framework to use for building charts - dashboard will integrate with Prometheus, as metric store - it is expected that only short term metrics will be supported (7 days) - python-observability-client will be used to query Prometheus Cross-project session with Nova team ============================= - sean-k-mooney led topics on how to evolve Nova to better assist other services, like Watcher, to take actions on instances. The team agreed on a proposal of using the existing metadata API to annotate instance's supported lifecycle operations. This information is very useful to improve Watcher's strategy's algorithms. Some example of instance's metadata could be: - lifecycle:cold-migratable=true|false - ha:maintenance-strategy:in_place|power_off|migrate - It was discussed that Nova could infer which operations are valid or not, based on information like: virt driver, flavor, image properties, etc. This feature was initially named 'instance capabilities' and will require a spec for further discussions. - Another topic of interest, also raised by Sean, was about adding new standard traits to resource providers, like PRESSURE_CPU and PRESSURE_DISK. These traits can be used to weight hosts when placing new VMs. Watcher and the libvirt driver could work on annotating them, but the team generally agreed that the libvirt driver is preferred here. - More info at Nova PTG etherpad [0] and sean's summary blog [1] [0] https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d [1] https://www.seanmooney.info/blog/2025.2-ptg/#watcher-topics Please let me know if I missed something. Thanks! -- Douglas Viroel - dviroel

4 7