April 2026 - openstack-discuss - lists.openstack.org

[nova][ptg] 2026.2 Hibiscus PTG summary
by René Ribaud 27 Apr '26

27 Apr '26

Hello everyone, Last week was the PTG, thank you to all who joined and contributed to the discussions! I hope you enjoyed it and found it productive. This cycle we had another packed agenda with active participation throughout the sessions. For the Nova sessions (Monday to Friday), we had an average of 15 to 20 participants per session. Attendance was slightly higher during the cross-project discussions, with an average of 25 to 31 participants, especially the Ironic session. You can revisit all notes and discussions on the Etherpad here: https://etherpad.opendev.org/p/nova-2026.2-ptg Below is a summary of the topics and outcomes from our PTG discussions. I've tried to keep it as concise as possible, though we covered quite a lot during the week. **** Monday topics **** #### 2026.1 Gazpacho Retrospective #### 2026.1 Gazpacho results: - Blueprints: 11 Accepted, 9.33 Implemented (84%) - Bugs: At least 28 bugfixes merged For comparison, 2025.2 Flamingo had 14 accepted blueprints with 9 implemented (64%) and at least 26 bugfixes merged. Release closure proceeded smoothly post-Feature Freeze with minimal CI issues. The only exception involved a pkg_resources issue outside Nova's control. The approach of accepting fewer, smaller tasks proved more predictable than previous cycles. ✅ Core +2 on a spec includes implicit commitment to reviewing the implementation. ✅ Upstream meetings should explicitly communicate priorities and readiness status. ✅ Bug tracking for eventlet removal work via Launchpad tags. #### 2026.2 Hibiscus Proposed Planning #### The team agreed on the following schedule: - Spec review day: May 21 - Soft spec freeze: May 28 - Spec review day: June 4 - Hard spec freeze: June 11 - M2: July 2 - Feature review day: August 4 - Feature review day: August 20 - Feature Freeze at M3: August 27 - Release: End of September / early October ✅ Adopt the proposed timeline, moving soft freeze two weeks earlier and scheduling review days one week before deadlines. ### Start cross session with Cinder ### #### Assisted Volume Extend #### The team discussed how to handle volume extend for file-based volume drivers (such as NFS) that require hypervisor assistance. Four options were analyzed (A through D), ranging from async callbacks to a generic assisted-volume-action API. ✅ Proceed with a hybrid approach combining Options A and B. ✅ Complete Cinder and client-side work first. ✅ Ensure required Nova API microversion availability during Cinder calls. ✅ Use OpenStack SDK (novaclient no longer receives support for newer API microversions). #### Expose Quiesce/Unquiesce API #### The team discussed creating a new API for quiesce/unquiesce instance volumes callable from Cinder, independent of snapshot operations. ✅ Create new API for quiesce/unquiesce instance volumes callable from Cinder. ✅ Support list of volumes for consistency groups. ✅ API should be asynchronous but with success/failure reporting capability. ✅ Require new specifications for both Nova and Cinder projects. ### End cross session with Cinder ### ### Start cross session with Manila ### Good progress was made since the last cycle. The SDK change and the OpenStack Client update have both merged. Testing has also progressed, with scenario test specifications merged and active work on Tempest modifications, virtiofsd installation in Manila devstack, and scenario tests for both VirtioFS attachments and hot attach. #### Hot Attach/Detach with VirtioFS #### Hot attach serves as prerequisite for live migration capability. The main discussion focused on whether a new API microversion is needed, since the request/response body doesn't change only the server-side precondition (allowing "active"/"paused" states instead of only "shutoff"/"error"). ✅ New microversion preferred as a feature availability marker, especially considering in-progress upgrade scenarios with mixed host states. #### Live Migration with VirtioFS Attachments #### The team discussed requirements (virtiofsd >= 1.11, QEMU >= 9.x, libvirt >= 10.x) and scope of blocked operations (unshelve, evacuate, rebuild, resize). ✅ Create two separate specifications for cold and live migration to decompose the workload. ✅ Cold migration absence represents an operator blocker that needs to be addressed. #### Memfd Support #### Adds hw:memory_backend flavor extra spec enabling per-instance shared memory backing selection (memfd/file/hugepage/anonymous). This enables OVS-DPDK and virtio-fs support, reducing the complexity. ✅ Make memfd the default immediately for new instances rather than a two-phase rollout. ### End cross session with Manila ### **** Tuesday topics **** ### Start cross session with Cyborg ### A more complete summary of this joint session will be provided by the Cyborg team, as they can better capture the discussions and decisions made. Reference: https://etherpad.opendev.org/p/cyborg-2026.2-ptg#L404 ### End cross session with Cyborg ### ### Start cross session with Ironic ### A more complete summary of this joint session will be provided by the Ironic team, as they can better capture the discussions and decisions made. On a lighter note, there was a historic and somewhat surreal moment during the session where Dan and TheJulia actually agreed on a topic, and Sean said something positive about OTU. 😄 ### End cross session with Ironic ### #### Unpinning Availability Zone #### The team discussed how to release instances pinned to specific AZs, preventing emergency migrations. Multiple approaches were considered. ✅ Allow setting the request_spec AZ to None or to the instance's current AZ to unpin. ✅ Default policy: admin-only. ✅ Document possible side effects of AZ unpinning. ✅ Preference for consistent behavior across all move operations. #### PCI Grouping #### Moving from "single device per alias" to "logical group of devices" is requested and scheduled as a single unit. Extensive discussion on traits (union approach), device types, PF/VF dependencies, resource class overrides, group lifecycle, and upgrade path. ✅ Union traits from grouped devices, devices within groups become non-consumable individually. ✅ Support multiple VFs from the same PF in different groups. ✅ Reject adding devices to allocated groups at compute startup. ✅ Handle device disappearance similarly to non-grouped devices. ✅ No reshape required; rolling upgrades supported. ✅ gibi, taking the primary implementation role, with Sylvain and Sean as main reviewers. #### Supporting Different Resource Classes for VFs (GPU/MIG Use Case) #### Modern GPUs (NVIDIA H100 with MIG) use SR-IOV to expose different compute profiles as VFs under a single PF. The current Nova implementation prevents assigning different Resource Classes to these VFs. ✅ Team not opposed, specification required. ✅ Coordinate with gibi to prevent PCI Grouping conflicts. ✅ Priority for this work needs to be defined. **** Wednesday topics **** #### Confidential Computing: AMD SEV-SNP Support #### Multiple parties report SEV-SNP running in production. The LY implementation is available and needs rebase on master. ✅ Start from LY implementation, refine via iteration. ✅ Takashi to update specification regarding hw_firmware_stateless definition. ✅ Avoid in-Nova workarounds for firmware; leverage libvirt detection. ✅ Documentation should describe firmware file modification requirements when needed. ✅ Omit idAuth/idBlock/hostData parameters from initial specification. #### Confidential Computing: TDX Spec Review #### Discussion covered TDX requirements (host passthrough, UEFI firmware limitations) and how to handle default memory-encryption model selection. The team really appreciated the quality of the presentation on this topic. ✅ Mandate hw:mem_encryption_model specification for TDX (not addressing default logic within this spec). ✅ Use Placement reservations for TDX key tracking rather than auto-detection. ✅ Document UEFI firmware limitation regarding SCSI driver stack. #### SEV Refactoring Remaining Issues #### The conflict between hw:mem_encryption=true and hw:locked_memory=False was discussed. ✅ Documentation patch to clarify interaction and expectations. #### Eventlet Removal #### Significant progress since Gazpacho. Remaining work includes console proxy conversion, CLI/nova-manage handling, deprecation statement, config option conversion, and full test coverage. ✅ Kamil volunteered to drive the Eventlet removal effort in Hibiscus. ✅ Explore generic eventlet import prevention (possibly via oslo). ✅ Verify novnc architecture in eventlet mode before conversion. ✅ Current Rally perf-test proposal represents the starting point; significant additional scenario work is needed. ### Start cross session with Horizon ### A more complete summary of this joint session will be provided by the Horizon team, as they can better capture the discussions and decisions made. Reference: https://etherpad.opendev.org/p/apr2026-ptg-horizon#L127 ### End cross session with Horizon ### #### Cross-AZ Scheduling Weigher #### The team discussed a weigher proposal for AZ-aware scheduling. Multiple concerns were raised about the approach. ✅ The pain point is understood, but the proposed weigher solution lacks fit. ✅ Alternative approaches involving server groups API or unified hard anti-affinity policies merit further exploration. ✅ No weigher adoption, topic requires further discussion. #### Add Typing #### The team discussed adopting type hints in Nova. Majority of dependencies now include type hints. ✅ Set ignore_errors = false for non-test code. ✅ Don't target strict typing. ✅ In conclusion, we have reached a consensus: type hints are welcome and should not be a reason for rejection. However, we reserve the right to push back if the hints provide no tangible benefit, the criteria for which will be refined over time. **** Thursday topics **** #### AI/LLM Knowledge and Hard Rules #### Discussion on adding knowledge prose and hard rules to Nova repository for LLM-aware development. ✅ Keep Nova knowledge in Nova repo without duplication. ✅ AGENTS.md should be minimal and point to in-repo documentation. ✅ Separate LLM-specific information from human-readable content. ✅ Prefer deterministic checks (scripts, linters, fixtures) over LLM prose guidance. ✅ Information useful to both humans and LLMs should live in documentation. ✅ Bring broader OpenStack governance questions about AI assistance to the TC and Board. #### Send os-server-external-events via SDK Instead of Novaclient #### Projects (Neutron, Cyborg, Cinder) send notifications to Nova via os-server-external-events API using novaclient. OpenStack SDK currently lacks this API. ✅ Add service-to-service API to SDK. ✅ Lajos to discuss with Stephen about sharing knowledge for similar API usage elsewhere. #### Bidirectional RPC Liveness Handshake #### Under certain conditions, Nova-Compute loses the ability to receive RPCs while the heartbeat continues, and the compute appears alive despite being unable to process requests. ✅ Specification accepted. ✅ Two-step implementation: unconditional first, then optimized. ✅ Failure threshold requirement; optimize later. ✅ Address timeout values, failure thresholds, and echo request pileup concerns in the spec. #### TempURL Usage in Nova #### The team briefly discussed the proposal for TempURL usage in Nova. ✅ General agreement on the approach. #### Attestation Integration #### Rather than Nova performing attestation, an external service will maintain host attestation traits. Nova auto-deletes these traits on startup, signaling unknown state. ✅ Define trait prefix (e.g., HW_ATTESTATION_*); Nova auto-deletes on startup. ✅ External service maintains traits at runtime. ✅ Flavors request/require traits for secure-node-only boot. ✅ Watcher may implement this via proxy service. #### Barbican Secret Handling for vTPM Live Migration #### Nova creates Barbican secrets for 'deployment' TPM security mode using Nova service user credentials. Barbican defaults allow any service project member to decrypt, which is broader than intended. ✅ Leave current design; operator responsibility. ✅ Melwitt to add documentation details/warnings/clarity to vTPM docs. **** Friday topics **** Note: Friday sessions were limited (13:00-14:00 UTC only) due to TC and Board meeting conflicts. #### Nova-Reviewers and Nova-Approvers Split #### Sean proposed adopting the Ironic/SDK model with separate nova-reviewers (+2) and nova-approvers (+W) groups. Extensive discussion followed on review bandwidth, core expansion, and how to increase patch landing velocity. The team debated multiple perspectives: - Expanding the reviewer base with proven subsystem experts - Aggressively adding full cores rather than creating a split - Concerns about lowering the landing bar ✅ No final decision reached, the team broadly agrees on the need to expand review capacity. ✅ Core expansion and mentoring remain the preferred path forward. ✅ Further discussion needed, potentially in private follow-up. #### Standardize Service-to-Service Communication #### We skipped this topic due to time and Gmann constraints. We will revisit this probably in an upstream meeting. #### Nova Ephemeral Volumes Using NetApp FlexGroup #### The team discussed a proposal for using NetApp FlexGroup as a backend for Nova ephemeral volumes. ✅ There are major concerns about the proposal. ✅ The approach looks somewhat hacky and could introduce technical debt in the future. A proper driver would be a better approach. ✅ Significant concerns were also raised about CI coverage. ✅ The team is not comfortable including a vendor-specific backend. Vendor neutrality will always be the preferred approach. ✅ A new spec proposal addressing these concerns will be welcome and discussed. #### Metadata/Tag Protection #### The team discussed the proposal for protecting instance metadata and tags from unauthorized modifications. ✅ Globbing preferred over regexp for pattern matching. ✅ Sean to leave notes in the spec and continue to refine it. ✅ Adding a concrete use case to the spec would be helpful to define the proper solution. If you've read this far, thank you! If you spot any mistakes or missing points, please don't hesitate to let me know. Best regards, René.

1 0

[nova][bug triage] Bug Triage Meeting - Tuesday April 29th
by René Ribaud 27 Apr '26

27 Apr '26

Hi all, Apologies for the late notice. The next Nova bug triage session will take place on Tuesday, April 29th. Here is the link to join: https://meet.google.com/fdz-phux-cnc As a reminder, this meeting is public and open to everyone. It's easy to join the roster. Feel free to add yourself: https://etherpad.opendev.org/p/nova-bug-triage-roster If you'd like to be added to the calendar invite for this meeting, feel free to ping me. Best regards, René

1 0

[cyborg] 2026.2 (Hibiscus) Virtual PTG Summary
by Sean Mooney 27 Apr '26

27 Apr '26

Hi all, As many of you may be aware the OpenStack community has a Project Team Gathering event every release cycle. I'm happy to report that the Cyborg team held its 2026.2 virtual PTG session last week and I would like to thank all who participated. Below is a summary of the topics covered, decisions made, and open questions where wider community input is welcome. Full notes are at: https://etherpad.opendev.org/p/r.bb485b94c3487349f2e36eb4e3b5a2ce Testing ------- A significant portion of the session was spent on the state of testing across the project. The conclusion is that coverage gaps are wide and need a structured effort to close. Driver support levels: We agreed to adopt formal driver support levels: supported, experimental, deprecated, and removed. All current in-tree drivers will be marked experimental immediately except the fake driver. Drivers that remain experimental for more than one release will be downgraded to deprecated after a mailing-list callout for maintainers. Cyborg will emit a start-up warning for drivers that are not at the "supported" level. As testing and documentation is built out for existing drivers they will graduate to supported over time. Exact details of the support levels will be captured in new project docs to set clear expectations for operators and developers. Once a deprecated driver has been announced in a SLURP release it can be removed in a following release. Tempest plugin: A dedicated etherpad will be created to coordinate cyborg-tempest-plugin improvements. Current gaps include missing stable test IDs, microversion tests, and per-driver scenario tests (today only one scenario test exists and it is hard-coded to the fake driver). PCI driver testing may be possible using the nvmevirt simulator or a third-party CI via RDO zuul. Functional testing: Building out functional test infrastructure (end-to-end tests within a single process against SQLite, similar to Nova's functional tests) was agreed as an aspirational goal for this cycle. A blueprint will be created; whether a full spec is needed is TBD. Contributor documentation will accompany it. python-cyborgclient: The v1 API code will be dropped this cycle; v2 Python bindings and the freestanding cyborg CLI will be formally deprecated in favour of the OpenStack SDK and OSC plugin. New test coverage will be built only around the OSC and sdk code path. Opens: Whether temporary operator-supplied test reports remain acceptable long-term and what minimum evidence they should contain; whether per-driver tempest tests become a hard gate for new drivers. Tech Debt --------- Several cleanup and scalability items were agreed for this and upcoming cycles. SRBAC: All phases will be implemented in 2026.2 (admin, manager, member, reader, and service roles). The new policies will become the default in 2027.1; legacy policies will be deprecated for removal in 2027.2, the exact details of this will be captured in a new spec. Conductor decoupling: Placement interactions should be moved to the compute agent; simple API CRUD operations should bypass the conductor RPC entirely. The conductor is currently a global single point of failure and scaling concern because cyborg does not support cells or conductor group for internal sharding of compute agents. The evolution of this will take time and thought and as such may be deferred to a later cycle. Quotas: The existing quota system is non-functional and will be replaced with Keystone unified limits. Cyborg must enforce quota on ARQ bind before device-profile on server create can be supported. Priority of this new capability is TBD and will partly depend on usage/operator feedback. Image verification: The feature was never fully implemented and depends on Nova objects that were never ported. Agreed to remove it and re-implement it from scratch with proper test coverage and documentation. With that said we may decide to fix it in place if that proves quick and back-portable. OpenStack SDK: Cyborg will port its glance interaction to the SDK, removing the python-glanceclient dependency. API design: The device, attribute, and deployables APIs have overlapping responsibilities and should be consolidated into a single surface. PATCH with JSON Patch payloads should be replaced with PUT. OpenAPI request and response schemas should be introduced. This cycle: produce a single design document. API evolution (microversion or v3) targeted for a future release. The only concrete change landing this cycle is removing stale references to the removed v1 API from the API reference. Documentation: New features must ship a release note and docs in the same series. New drivers must include a driver-specific admin doc and a cross-link to the support matrix. We also agreed to produce a service interaction doc covering how cyborg, placement, and nova work together. Opens: How to signal enabled/disabled device state to the Nova scheduler. reserved=total vs placement traits both have precedent but different trade-offs around admin reservations and per-resource granularity. Using reserved=total is likely the correct path but we need to document why and how that works. Nova Cross-Project ------------------ We met with the Nova team to align on work that requires changes in both projects. VM lifecycle (resize, cold/live migration): Gaps for stateless devices will be investigated this cycle with a PoC if time allows. The feature is targeted at 2027.1. Stateful device support (e.g. NVMe disk content transfer) is deferred to 2027.1+. For resize-revert, the expectation is that the source ARQ is held until the resize is confirmed and the destination ARQ is held until a revert is discarded via resize confirm. Attachment handles: An mdev handle type is needed for time-sliced vGPU support (Nova spec already in review). Handle content will need to carry live_migratable and managed metadata that would normally come from the nova.conf via the PCI devspec. Additional future types (nvme, block, usb, cxl) are scoped but not targeted this cycle. Device-profile on server create and attach/detach: Both have a soft dependency on unified limits support landing in cyborg first. Targeted at 2027.1+ after resize is supported. Nova scheduler extensibility: Not discussed due to time. The MVP proposal is to expose provider summaries to existing filters and weighers via the host state object. Longer-term proposals for stevedore extension points and hook points were mentioned as nice to have operator quality of life improvement for out-of-tree use cases (watcher policy-based scheduling, cyborg NUMA filtering, telemetry-aware scheduling). Opens: Whether a generic configurable traits weigher in Nova, or moving weighing logic to Placement, could satisfy the extensibility use cases without requiring out-of-tree hook infrastructure is still open. New Drivers ----------- Two new drivers are moving forward this cycle alongside early groundwork on a longer-term v2 driver framework. Generic MDEV driver: The existing NVIDIA driver will be refactored to reuse it. First-party CI testing will use the mtty/mdpy kernel modules. The companion Nova spec for mdev attach-handle support is already in review. NVMe driver: It will inherit from the generic PCI driver, use nvme-cli for device management and secure erase, and will not require the pci or SSD driver to be enabled to function. Details will be in the sepc. Generic driver framework v2: Agreed to PoC the concepts incrementally in the new MDEV and NVMe drivers this cycle, factoring out PCI enumeration and introducing typed data classes in place of generic dicts. No spec this cycle; a formal spec targeting 2027.1 will follow if the PoC succeeds. We also agreed to rename the whitelist config option to align with Nova's devspec so that shared keys behave consistently across both projects. Placement traits and RP naming: The convention CUSTOM_<BUS>_<vendor_id>_<product_id> (aligning with Nova's PCI-in-placement work) was proposed and discussed but no formal agreement was reached this cycle. Interchangeable VFs should be grouped into a single RP per NUMA node per device type rather than reported as individual RPs of inventory 1. A spec with an upgrade path is required before any of this lands. TBD if this will be done as part of the v2 generic driver work or independently; no ETA is proposed for this work. Opens: NUMA modelling in placement requires agreement with Nova on the RP naming convention and UUID generation before cyborg can create NUMA resource providers independently. If you have questions or feedback on any of the above, please reply to this thread. Thanks to everyone who participated in the sessions. regards Seán p.s. as with most of my longer prose to this list this has been translated form sean speak to english via llm assistance, enjoy the added punctuation and capital letters.

1 0

[watcher][ptg] Watcher 2026.2 Hibiscus PTG Summary
by Douglas Viroel 27 Apr '26

27 Apr '26

Hi all, Thank you all that joined the Watcher PTG sessions last week. We had productive discussions across three days covering technical debt, new features, scalability improvements, and testing priorities. The Watcher PTG etherpad used during the sessions can be found here[1]. Below is a summary of the key topics and decisions. Eventlet Removal (dviroel) ==================== We reviewed progress on the community-wide eventlet removal effort. Native threading mode with environment variable support was added in Flamingo, and all Watcher components now support native thread mode. We merged the patch making native threading the default for 2026.2. For remaining tasks, we agreed that new scaling tests will provide updated performance results now that threading is the default. We discussed the need for performance testing but determined that the rally job may not be ready for this cycle. Instead, we plan to leverage emulators or fake drivers for performance testing. *Agreement*: Eventlet code removal will happen early in the 2027.1 release cycle (around November 2026), including removal of eventlet CI jobs and adding checks for eventlet imports. Telemetry and Datasources ===================== a) Prometheus and Aetos Support (dviroel) We had a cross-team session with the Telemetry team to discuss datasource status. Aetos support was added in Flamingo, and Prometheus was deprecated in Gazpacho. We discussed the timing of Prometheus code removal and agreed to defer it to align with other projects like Cloudkitty and Aodh that still support both datasources. We identified the would be good to add a lightweight Aetos job to devstack-plugin-prometheus changes and also to test Application Credentials with Aetos in devstack jobs. *AI*: Work on an Aetos job to run in devstack-plugin-prometheus changes. *AI*: Test Application Credentials with Aetos in devstack jobs. b) New Metrics for Watcher (dviroel) We discussed new metrics that can be consumed by existing strategies and enable implementation of new strategies. Key metrics identified include: - Network throughput metrics (current ceilometer provides packet counts) - Disk latency (can be calculated from ceilometer's read/write request counts and times) - CPU steal time (vcpu.delay from libvirt domstats) for identifying contention in noisy neighbor scenarios. This metric is not available in ceilometer today. - Host-level CPU/memory/IO pressure (PSI metrics via node_exporter) We confirmed that new metrics can be added to Ceilometer without proposing a spec. The Telemetry team can help identify the effort required once we raise specific requirements. *Agreement*: Provide more details in a blueprint and present it at weekly meetings in the IRC. Cluster Data Model Extensions (amoralej/dviroel) ===================================== We discussed the addition of new attributes to the Cluster Data Model (CDM). Recent additions include pinned_az and flavor extra specs. With the openstacksdk migration complete for nova integration, we can now retrieve scheduler_hints and image_properties from the Nova API. New attributes under consideration for servers include: - cpu_info (to identify different architectures) - OS-EXT-STS:task_state, OS-EXT-STS:vm_state, instance created_at (needed for zone_migration strategy to remove Nova dependency). For compute nodes, attributes like hypervisor_type, hypervisor_version, and CPU architecture would help identify compatible destination nodes for migrations. We reached important agreements on the datamodel list API: *AI*: Update Watcher documentation to reflect that the datamodel list API is frozen. *AI*: Define and document the deprecation process for CDM attributes in the Contributors Guide, as removing attributes may affect third-party strategies. Filters for Watcher Strategies (dviroel) ============================= We discussed how to leverage CDM attributes to provide better solutions through filters that assist strategies in selecting valid destination nodes. Filters would apply to static information (capabilities, node and instance attributes, scheduler hints, server groups, availability zones) but not to metrics, which each strategy evaluates independently. The discussion covered whether to implement filters as a plugin-based architecture using stevedore, though we noted that the plugin architecture is already available for strategies themselves. Multiple filters could be chained, and selection could be controlled via configuration options. *Agreement*: Further investigation and refinement of the strategy filtering destination functionality. A PoC is desired and a spec proposal is needed. Watcher Dashboard Improvements (chadankumar) ======================================= a) Django 5 Migration We reviewed the Django 5 migration status. The horizon-tox-python3-django52 job is passing but doesn't test watcher-dashboard due to missing test coverage. We agreed that we need playwright tests running with both Django 5.2 and older Django versions. b) User Flows and Playwright Testing We identified key user flows to test with Playwright integration testing: - Creating dummy audits, viewing action plans and actions - Skipping actions and starting action plans - Audit scope workflow testing with custom parameters - Archiving action plans, audits, and actions - Continuous audit flows - Event-based audit flows (pending tempest plugin work) *AI*: Review the proposed architecture and patterns proposed for watcher-dashboard. *AI*: Review watcher-dashboard patches to allow new Playwright tests to be implemented. CI Improvements (chadankumar) ========================= a) Grenade Testing We agreed to move the watcher-grenade job to multinode configuration to properly run scenario tests, as all scenario tests require two nodes. The multinode grenade job only upgrades the controller node. b) Testing Coverage Gaps We identified high-priority features missing testing coverage: - **Audit scope** (marked as must-have, very important feature) - HIGH PRIORITY - **Event-type audits** - HIGH PRIORITY - **Baremetal integration** (Ironic) - LOW PRIORITY For Ironic integration, we need to check the feasibility of getting a CI job simulating Ironic nodes using a fake driver or vbmc. Given the lack of documentation, testing, and use cases, we agreed to deprecate the Ironic integration in this release with removal deferred to future releases. *AI*: Propose new watcher-tempest-tests for audit scope. *AI*: Propose new watcher-tempest-tests for event-type audits. *AI*: Check how feasible it is to get a CI job simulating Ironic nodes using a fake driver or vbmc. c) Simulator Improvements We discussed the Alfredo's simulator tool under development [2], agreeing it can be incorporated into the project. We identified the need to improve it by generating topology and metric files instead of using large static files which are harder to review and maintain. *Agreement*: Propose improvements to the simulator as discussed at the CI improvements session. Datasources, Integrations, and Strategies Status Review (dviroel) ================================================== a) Datasources Review - **Monasca**: Removed in Gazpacho - **Prometheus**: Deprecated in Gazpacho; code removal deferred (see Telemetry section) - **Grafana**: Missing testing and documentation; we agreed to deprecate it in 2026.2 *AI*: Propose Grafana deprecation patch and send email to the mailing list regarding Grafana deprecation. b) Integration Status - **MAAS**: Deprecated in Gazpacho; agreed to remove code in Hibiscus as it has eventlet code requiring refactoring - **Ironic**: Missing documentation and testing; deprecate in this release (see CI Improvements section) - **Glance and Neutron**: Removed in Gazpacho; documentation needs updates *AI*: Propose patch for MAAS code removal. *AI*: Propose Grafana deprecation patch and send email to ML calling out for maintainers. c) Strategy Removals We discussed the removal of the deprecated noisy_neighbor strategy and goal. We agreed that we need a "soft-delete" approach where deprecated strategies are blocked from new audits but existing audits using them can still be archived during upgrades. This prevents issues with the database migration. *AI*: Propose a plan for removing a strategy and a goal from Watcher. Document it in the Contributors Guide. d) Strategies Missing Tempest Coverage - **saving_energy**: Depends on node power state change action and Ironic integration - **storage_capacity**: Tempest test is being skipped; has tech debt with direct Cinder API calls (https://bugs.launchpad.net/watcher/+bug/2142219) - **uniform_airflow**: Requires host_airflow, host_inlet_temp, host_power metrics - **outlet_temperature_control**: Requires host_outlet_temp metric *AI*: Understand what is missing to get storage_capacity test running in CI and propose a fix. *AI*: Work on storage_capacity tech debt: https://bugs.launchpad.net/watcher/+bug/2142219 *AI*: Investigate whether airflow, temperature, and host power metrics are available in currently supported datasources/exporters. OpenstackSDK Migration (jgilaber) ========================== The Nova helper migration is complete. We plan to tackle Keystone, Placement, Ironic, and Cinder using the same pattern applied to Nova helper. The first patch for each helper will remove unused methods. The openstacksdk fix for retry configuration[3] was merged and released in openstacksdk 4.11.0. We will revisit the Ironic integration migration given the deprecation discussion. *Agreement*: jgilaber will continue proposing the migration patches during this release. Patches will focus in one service at a time. Scalability Topics (amoralej) ===================== a) Limiting Action Plan Size Consistently We discussed defining a mechanism to limit action plan size consistently across strategies. We agreed on a two-level approach: - System-wide config settings provide hard limits with default values - Audit parameters allow users to customize limits (e.g., number of hosts evacuated, total VM migrations) Different strategies have different applicable limits: - Consolidation strategies: number of hosts evacuated, total VM migrations - Workload stabilization: total VM migrations - Zone migration: total VM migrations, VM migrations per host, total volumes, volumes per backend Parameter names should be consistent and common across strategies, possibly grouped by resource type. One alternative is to use Audit Scope to reduce the size of the solution and, respectively, the size of the Action Plan, but it may not solve the problem for all use cases. In zone_migration strategy, existing parameters would be translated to new ones to maintain compatibility. *Agreement*: Design the implementation and provide documentation or a spec for the team to review. b) Parallelization of Action Execution We discussed providing a way to define parallelization of actions in action plan execution for all strategies. Currently, parallelization is defined at two levels: - In the Applier: limited by config value max_workers - In the audit execution: set by the planner (varies by strategy) We discussed criteria to optimize action plan order common to all strategies: - Parallelization level per action type - Interleaving migrations based on source_node when planning migrations - Pipeline of distinct groups which must be optimized We agreed that system configuration should define this initially, not exposing it to users. The implementation should consider whether a common planner can be used or if per-strategy planners are needed. The team also mentioned the possibility of replacing current implementation of one threadpool per Action Plan by a single shared threadpool to make easier for user to configure limits. Another improvement mentioned for the Applier is the dispatch of Actions instead of Action Plang among the running Appliers. *Agreement*: Design the implementation and provide documentation or a spec for the team to review. Migration Action Prioritization and Filtering (amoralej) ======================================== We briefly discussed implementing a mechanism to prioritize, filter, and order migration actions in action plans that could be reused across audits. Use cases include: - Not migrating instances marked with specific metadata, tags, or hints - Prioritizing migrations based on instance characteristics (CPU count, memory size, metrics) - Applying ordering criteria to optimize execution time (e.g., round-robin distribution based on source compute node) We identified two points where this feature can be applied: - **Strategy execution**: Where filtering must be done to avoid breaking efficacy indicators - **Strategy planner**: Where ordering may be applied for parallelization Examples of filters: - Do not migrate VMs with certain metadata in Nova (e.g., "gold" or "untouchable") - Do not migrate VMs on compute nodes marked in Placement *Agreements*: These mechanism were partially discussed in other topics, so we agreed that we should revisit this in future discussions. Audit Pipeline (previous Strategy Stacking) (dviroel) ======================================= We wrapped up discussion on the audit pipeline feature design, covering the PipelineHandler, Audit Pipeline resource, mutable CDM, metric caching process, and planner optimizations. We briefly covered some outputs from a proof of concept made with a audit pipeline of 2 strategies[4]. Key agreements: - Use of nullable strategy_id field to enable dropping it later. - Minimum 2 stages per pipeline, maximum configurable via config option. - For CONTINUOUS audit pipelines, changing some specific parameters will be supported initially; changing stages themselves may not be supported yet. The spec needs to document mutable fields clearly *AI*: Update audit pipeline spec[5] to reflect recent comments and agreements from the PTG session. Workload Profiles and Tiers (dviroel) ============================ We discussed methods to identify workloads with different priorities that can be used by different/existing strategies. Two main approaches were identified: 1. **Flavor extra-specs**: Using keys like `optimize:priority` or `optimize:tier` (gold/silver/bronze) - Can be used by strategies to select low-priority VMs first - Can be used in audit scope to filter instances by priority/tier - Should be aligned with Nova's usage of extra-specs to avoid overloading their meaning 2. **Instance metadata**: Using Watcher custom keys like `watcher-priority` metadata - Approach used in Noisy Neighbor strategy, but is more unlikely be adopt by new strategies. The information would be available in the Compute Cluster Data Model for possible identified use cases: - Audit scope attributes to exclude/include instances based on profiles or priorities - New SLA enforcement strategy with metrics enforced based on performance profile. Today this is the main use case for this feature. *Agreement*: We agreed that any full proposal should include a real use case and be applicable in existing strategies or propose a new strategy/goal. Watcher can support multiple ways of annotating instances with overriding rules for flexibility. Preemptable Instances (winiciusallan) ============================= We discussed a new feature for preemptable instances that would allow Watcher to preempt/shelve instances to achieve specific goals. Use cases include: - Public clouds offering instances at lower prices using spare capacity - Private clouds prioritizing workloads such as HPC or AI jobs The high-level implementation concept includes: - Flavor/server property (e.g., `lifecycle:preemptable=true|false`) to identify preemptable instances at creation time - New Goal and Strategy for preemptable instances - New action(s) for delete or shelve operations - Threshold definitions for metrics to be considered - Policies for deletion vs shelving behavior This builds on previous community discussions and implementations like CERN's Aardvark (triggered by NoValidHost) and NeCTAR-RC's similar service. *AI*: Work on a use case and propose a solution using Watcher, including a spec with the feature design. Gazpacho Retrospective =================== We conducted a retrospective for the Gazpacho release: **What Worked Well:** - Weekly IRC meetings held consistently throughout the release - 2 new core reviewers added - Review feedback within a week - No untriaged bugs in backlog with improved triaging process and tagging - Refactored common code in watcher-dashboard and watcher-tempest-plugin - Good testing and analysis on scaling issues - Better dashboard testing tooling (focus on Playwright) - Better deadline management with less hurried merges - Documentation improvements - Adopted ruff for code modernization **What Needs Improvement:** - Stable branch maintenance and release cadence - Missing tempest coverage: strategies, audit scope, integrations, event-type audits - Functional tests *AI*: Include checkpoints in the current release so the team can review the status of stable backports, and release-liaisons can report on latest stable releases. Thanks again to everyone who participated! Douglas Viroel (dviroel), assisted by Claude Sonnet 4.5 [1] https://etherpad.opendev.org/p/watcher-2026.2-ptg [2] https://review.opendev.org/c/openstack/watcher/+/980257 [3] https://bugs.launchpad.net/openstacksdk/+bug/2142571 [4] https://etherpad.opendev.org/p/watcher-audit-pipeline [5] https://review.opendev.org/c/openstack/watcher-specs/+/969840 -- Douglas Viroel - (irc: dviroel)

1 0

[designate][neutron] can't release floating ip
by taaviansperr＠gmail.com 27 Apr '26

27 Apr '26

Hi I created two bug reports and asked in IRC. Neutron folks forwarded me to designate folks somewhat Any ideas on this issue? https://bugs.launchpad.net/designate/+bug/2150205 https://bugs.launchpad.net/neutron/+bug/2149807 Taavi

3 2

[PTG] #eventlet-removal 2026.2 Hibiscus PTG Summary - Eventlet Removal Initiative
by Herve Beraud 27 Apr '26

27 Apr '26

Hi everyone, Following our office hours session at the Hibiscus PTG, I'd like to share a brief summary of our Gazpacho retrospective, the key achievements we've celebrated, and the priority actions we've identified for the Hibiscus cycle. ## Gazpacho Retrospective The Gazpacho cycle marked a significant turning point in our eventlet removal journey. Three projects reached full migration completion: Designate, Neutron, and Cyborg joined the ranks of fully migrated projects alongside Octavia, Mistral, Ironic, Heat, and Barbican. This brings us to eight major projects that have completely removed their eventlet dependencies. Nova made remarkable progress with five out of seven services now defaulting to threading mode. The API service, metadata API, and scheduler switched to threading by default in Gazpacho, while compute and conductor followed suit in early Hibiscus. Watcher achieved a similar milestone by switching all components to threading mode by default. Manila developed a reusable Rally-based performance testing framework that demonstrated no significant performance differences between eventlet and threading modes, providing crucial validation for the entire initiative. On the infrastructure side, the eventlet library itself now fully supports Python 3.14 stable after resolving compatibility issues with asyncio. Oslo.service introduced multiprocessing spawn support to reduce fork instability in multi-threaded environments, particularly benefiting Manila deployments. ## Hibiscus Call to Actions The Hibiscus cycle represents a critical phase where our focus shifts from initial migration to defaulting projects to threading mode and addressing remaining technical blockers. Nova will prioritize completing the console proxy migration for noVNC, spice, and serial consoles, which remain the last services pending threading support. Projects that have completed their core migration work should now turn their attention to converting functional tests to threading mode and expanding test coverage. With Ubuntu 26.04 releasing on April 29, 2026, we need to enable Python 3.14 testing across all projects to validate our compatibility work. This testing will be essential for ensuring smooth deployments on the new LTS release. Projects should also begin defaulting their services to threading mode following the recommended three-release pattern: one cycle supporting both modes with eventlet as default, the next cycle switching to threading as default, and the final cycle removing eventlet support entirely. Library maintainers should focus on cleaning up unconditional eventlet imports across Oslo libraries and making eventlet dependencies truly optional. This cleanup work will unblock projects from declaring themselves fully eventlet-free even when they depend on Oslo components. ## Critical Concerns Requiring Attention Three critical issues emerged during our discussions that need immediate community focus. The picklable problem affects multiple projects including Nova and Neutron. Objects passed to multiprocessing spawn must be picklable, but config objects from Oslo.config currently are not. Python 3.14 changes the default multiprocessing context to fork_server, and the fork context is being deprecated, making this issue increasingly urgent. Root wrap compatibility in threading mode remains unclear and potentially blocks projects like Nova and Cinder through their OS-brick dependency. We need coordinated testing and potential workarounds or alternatives to ensure this critical privileged operation mechanism works correctly in the new threading model. The heartbeat performance regression issue has a clear solution path: reverting the problematic patch that introduced a half-second sleep in threading mode. The community agreed to proceed with this revert and explore TCP keepalive as an alternative, which already works successfully for Tooz and Oslo.messaging. ## Performance Validation and Testing Insights One of the most encouraging outcomes from Gazpacho was Manila's performance testing work. Their Rally comparison framework runs identical workloads on both eventlet and threading deployments, generating comparative reports that show no significant performance differences. This framework is adaptable for other projects and provides the concrete validation needed to build operator confidence in the migration. Swift faces unique challenges with erasure coding that requires predictable request ordering for fragment reconstruction. The team is carefully distinguishing between actual requirements for deterministic behavior and test artifacts that assumed eventlet's specific co-routine switching patterns. This work will provide valuable insights for any project dealing with complex ordering requirements. ## Timeline and Long-term Vision The community reaffirmed the timeline for complete eventlet removal. By 2027.1, all projects should support threading mode with threading as the default configuration, though eventlet may still be available for operators who need it. The 2027.2 release will focus on removing eventlet from Oslo libraries and cleaning up all transitive dependencies, marking the completion of the initiative. After eventlet is fully removed, the community will evaluate whether async.io adoption makes sense for specific use cases. The consensus is clear: we don't want to run three concurrent runtimes simultaneously, so async.io consideration must wait until eventlet is completely eliminated. For a comprehensive analysis including detailed project status, technical discussion transcripts, and the complete list of concerns and solutions, please visit the full report at https://removal.eventlet.org/guide/openstack/hibiscus-ptg/. The report also includes the complete PTG session recording for those who want to review the discussions in detail. Thank you to everyone who participated in the PTG session and continues to drive this critical initiative forward. The progress we've made demonstrates that full migration is not only possible but achievable on a reasonable timeline. Let's maintain this momentum through Hibiscus and continue moving toward a fully threading-based OpenStack. As always, you're welcome to join us on IRC at #openstack-eventlet-removal, and you can find guidance, resources, and updates at https://removal.eventlet.org/ Best regards, Hervé -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud

1 0

[nova] Proposals for changes to nova-core
by Stephen Finucane 24 Apr '26

24 Apr '26

[BCC current nova core group, not all of whom were present today] o/ During today's nova PTG sessions, we discussed Sean Mooney's recent proposal [1] to introduce new nova-approver and nova-reviewer groups, akin to the model used by Ironic in recent releases. The conversation was initially expected to focus on the onboarding of new nova cores, but ended up turning into a discussion on how best to use the experience of SMEs from other OpenStack services with integration in nova to make progress on their features in a timely manner. This was heavily inspired by discussions between the Ironic and Nova teams earlier this week. During the discussion, it became clear that there were a variety of potential options on the table and many of the Nova cores present had different and often competing preferences. In order to break the deadlock, we agreed to codify four proposals into changes to the Nova contributor docs submitted to Gerrit. This is now done, and the four proposals can be found here: * https://review.opendev.org/c/openstack/nova/+/986141 (option 1) * https://review.opendev.org/c/openstack/nova/+/986142 (option 2) * https://review.opendev.org/c/openstack/nova/+/986143 (option 3) * https://review.opendev.org/c/openstack/nova/+/986144 (option 4) I have attempted to capture the intent of the proposals in the changes themselves, but for the avoidance of confusion I will repeat the guidance here, in a bullet point format: * These patches *should not* be merged yet. Nova cores are asked to +1 the options that they are *okay with* and -1 those that are *not okay with*. A +0 vote is considered neutral and is not included. This is known as "Approval Voting". * Small issues with wording and typos can be noted, but should not affect the overall vote: we can address these issues before we merge the patch. * Additionally, none of these proposals should imply a change in our general policy for onboarding new cores. * We will proceed with the most popular proposal for the remainder of the H cycle and evaluate the outcome during the next PTG (expected in October 2026). At that point, we will decide whether to persist with the proposal and merge this patch, or make changes. The ask is for all existing nova cores to weigh in on the proposals and explicitly submit a -1, 0, or +1 vote on each patch. I will stress again that we are not looking for a solution that is everyone's favoured solution as such a solution does not appear to exist. Rather, we are looking for a solution that is *acceptable* to the largest percentage of existing nova cores. I propose that we give it a week for people to assess the options and provide their opinions. At that point we'll proceed with the chosen proposal for the duration of the H cycle. Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

1 0

[ops] Ops Radio Hour
by Chris Morgan 24 Apr '26

24 Apr '26

Hello Everyone, We did another Ops Radio Hour today at 1300 UTC. Some notes were captured on the long-running etherpad ( https://etherpad.opendev.org/p/openstack_ops_radiohour) These get recorded and posted to youtube through the diligence of Ildiko Vancsa and other foundation staffers. Ops Radio is an attempt to bring the OpenStack operators community together regularly in a cheap and cheerful way to discuss mutual concerns (we used to arrange in-person Ops Meetups but since the pandemic that became simply too difficult to achieve), but most people with an interest in such things can spare an hour for what is basically just another zoom meeting. One major challenge is the fracturing of the communications channels across many outlets - reddit, linkedin, irc, slack, matrix, mastodon, bluesky, x etc. As the community of OpenStack operators is not a single team in even the loosest sense, it's hard to have any kind of policy on how to talk to each other. A quick poll of those attending suggested this mailing list is once again a leading choice and hence here I am trying to share what we learned and next steps. - Next Ops Radio is currently pencilled in for Friday May 29 - Ongoing topics include : -- barriers to upstream contribution -- legacy operators docs (https://docs.openstack.org/operations-guide/ https://docs.openstack.org/arch-design/) which should be perhaps be officially retired in favor of pointers to current best practices in the various active silos such as OOO, Kolla, openstack-ansible etc. These were discussed today -- There was also a prior suggestion to discuss OpenStack Observability but those attending today didn't have comments on that. -- other eternal challenges include day 2 challenges (openstack cluster maintenance and upgrades), high availability. performance tuning, problems of scale and so on. -- !!your topic here!! - feel free to drop suggestions for discussion on the agenda on the above-linked etherpad. We would also welcome guest speakers and additional host/moderators. Cheers Chris Morgan -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[i18n] Summary of 2026.2 Hibiscus PTG Discussions
by Ian Y. Choi 24 Apr '26

24 Apr '26

Hello all, I would like to share a summary of the recent discussions held during the 2026.2 Hibiscus virtual PTG regarding the i18n SIG. We focused on advancing the Weblate migration through Zuul job testing and language configuration challenges, alongside updates on AI-assisted translation pipelines from the KNU team. Special thanks to all attendees including the contributors, OpenDev infra team, and KNU students for their contributions. Please refer to the full details from our Etherpad: https://etherpad.opendev.org/p/apr2026-ptg-i18n 1. Zuul Job Development for Weblate Migration We reviewed progress on testing Zuul jobs for Weblate integration using scripts upstream_translation_update_weblate.sh and propose_translation_update_weblate.sh. - Current Status: Scripts verified locally and now under Zuul testing with Contributor Guide and Horizon projects; test environment built via [1]. - Next Steps: Virtual sprint proposed for Zuul implementation around June 3 (14-16 UTC), with infra team availability confirmed [2]. 2. Language Migration from Zanata to Weblate Discussions centered on applying Zanata language settings to Weblate and handling discrepancies. - Progress: Python script for settings application (Gerrit #961371) [3]; following OpenInfra Korea Forum process [4]. - Challenges: Need scripts for zanata.json and zanata-plural.json recreation; plural mismatches in languages like as, bn_IN, gu, kn, pt, ru, tr_TR. - Tasks: Develop JSON generation scripts and decide plural handling strategy. 3. KNU Team AI Translation Pipeline Wrap-up KNU students shared their PoC for AI-based draft translations on new POT entries. - Accomplishments: Ansible role (prepare-ai-translation) with 4-step CI pipeline using Ollama; supports 44 languages, preserves human translations, integrates OpenStack glossary. - Repo: [5]. - Remaining: Zuul feasibility for Ollama (consider llama.cpp); not yet tested in Zuul. 4. Additional Topics - Future of Zanata infra (Docker with JVM?); OpenStack-Helm translation support; 14 open reviews in openstack/i18n [6]. - Proposals: Testing infra guidelines, I18n contributor guidelines; learn from Kubernetes SIG Docs L10n [7]; LLM on Zuul feasibility. Thank you, /Ian [1] https://github.com/ppiyakk2/zuul-i18n-config [2] https://wiki.openstack.org/wiki/VirtualSprints [3] https://review.opendev.org/c/openstack/i18n/+/961371 [4] https://forum.openinfra-kr.org/t/openstack-weblate-migration-language-migra… [5] https://github.com/openstack-kr/knu_i18n_2025 [6] https://review.opendev.org/c/openstack/i18n/+/981694 [7] https://mentorship.lfx.linuxfoundation.org/project/c71fdb2a-8e77-447d-ae8b-…

1 0

[blazar] Meeting schedule updates
by Matt Crees 24 Apr '26

24 Apr '26

Hi all, Due to members of the core team being unavailable, I am cancelling our meeting on 30th April. Given the clocks have changed, we've also agreed to move the meeting slot an hour earlier until the next time the clocks change. Our next meeting will now be 14th May at 2pm UTC. Cheers, Matt Crees (Blazar PTL)

1 0