Hello everyone,

Last week was the PTG—thank you to those who joined! I hope you enjoyed it.

I haven’t gathered exact attendance stats, but it seemed that most sessions had at least around 15 participants, with some peaks during the cross-team discussions.

If you’d like to take a closer look, here’s the link to the PTG etherpad: https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d

We had a pretty full agenda for Nova, so here’s a summary I’ve tried to keep as short as possible.

#### 2025.1 Epoxy Retrospective ####

17 specs were accepted, and 12 implemented — an excellent ratio. This represents a clear improvement over previous cycles.
Virtiofs was successfully merged, unblocking other work and boosting contributor motivation.

✅ We agreed to maintain regular status updates via the etherpad and follow up during Nova meetings.

API Microversions & Tempest Coverage, several microversions were merged with good structure.
However, some schema changes were not reflected in Tempest, causing downstream blockers.

Also the updates covered by the microversions were not propagated into the sdk and openstack client.

✅ Ensure client-side features (e.g., server show) are also published and tracked.
✅ Keep microversions isolated and document Tempest implications clearly in specs.
✅ Raise awareness of the tempest-with-latest-microversion job during Nova meetings.
✅ Monitor OpenAPI efforts in Nova, which may allow offloading schema checks from Tempest in the future.

Eventlet Removal, progress is behind schedule, especially compared to other projects like Neutron.

✅ Flag this as a priority area for upcoming cycles.

Review Process & Tracking, spec review days were difficult to coordinate, and the status etherpad often outdated.
✅ Encourage active contributors to support occasional contributors during review days.
✅ Commit to keeping the etherpad current throughout the cycle.

#### 2025.2 Flamingo Planning ####
Timeline:

    Soft spec freeze (no new specs): June 1st
    Hard spec freeze (M2): July 3rd
    Feature Freeze (FF): August 28th
    Final release: late September / early October

✅ We agreed to officially adopt June 1st as the soft freeze date, based on the successful approach in Epoxy.
✅ A spec review day will be scheduled around mid June, these will be scheduled and announced early to ensure participation.
✅ Uggla will update the schedule document with the proposed milestones.

#### Upstream Bug Triage ####

We acknowledged that active bug triage has slowed down, resulting in a backlog increase (~150 untriaged bugs).
There is a consensus that triage remains important to maintain a clear picture of the actual bug landscape.

✅ Trial a new approach: review some untriaged bugs at the end of Nova team meetings.
✅ Process the list by age (starting with the newest or most-voted first).

#### Cloud Hypervisor Integration ####

There is an ongoing effort to integrate Cloud Hypervisor into Nova via the Libvirt driver:
Spec: https://review.opendev.org/c/openstack/nova-specs/+/945549

The current PoC requires only minor changes to work with Libvirt, and the team is ready to present the proposal at the PTG.

✅ We’re happy with the direction the spec is taking. Below are some key highlights regarding the spec.
✅ Clarify platform support (e.g., is libvirt compiled with cloud hypervisor support by default? Is it available in distros?).
✅ Provide a plan for runtime attach of multiple NICs and volumes.
✅ Mark as experimental if cloud hypervisor is not yet in upstream distro packages.
✅ Ensure that the following features are expected to work and covered in the spec: resize, migrate, rebuild, evacuate, snapshot.
✅ Justify raw-only image support, and outline the path to qcow2 compatibility.


#### vGPU (mdev) and PCI SR-IOV Topics ####

1. Live-migratable flag handling (physical_network tag)
Bug: https://bugs.launchpad.net/nova/+bug/2102161

✅ We agreed that the current behavior is correct and consistent with the intention:
If live_migratable = false → fallback to hotplug during live migration.
If live_migratable = true on both source and destination → prefer transparent live migration.

✅ Investigate how Neutron might participate by requesting live-migratable ports.

2. Preemptive live migration failure for non-migratable PCI devices
Nova currently checks for migratability during scheduling and conductor phases. There’s a proposal to move these checks earlier, possibly to the API level.
Bug: https://bugs.launchpad.net/nova/+bug/2103631

✅ Confirm with gmann whether a microversion is needed — likely not, as return codes are already supported (202 → 400/409).
✅ Uggla may submit a small spec to formalize this change.
✅ Split the work into two steps:
- Fix existing bug (can be backported).
- Incrementally move other validations earlier in the flow.

3. PCI SR-IOV: Unify the Live Migration Code Path

#### SPICE VDI – Next Steps ####

There is an ongoing effort to enhance libvirt domain XML configuration for desktop virtualization use cases (e.g. SPICE with USB and sound controllers). Some patches were proposed but not merged in time for Epoxy.
Mikal raised the question of whether a new spec would be required in Flamingo, which would be the third iteration of this work.
The team also raised concern about the complexity of adding traits (e.g. os-traits) for relatively simple additions, due to the multi-step process involved (traits patch, release, requirements update, etc.).

✅ Proceed with a specless blueprint.
✅ Plan to pull os-traits and os-resource-classes logic into Placement, to simplify the integration process and reduce friction. Link the required Placement version in Nova documentation accordingly. This is a strategic direction, even if some traits might still be shared with Neutron/Cinder.

#### Virtiofs Client Support ####

The virtiofs server-side support was merged in Epoxy, but SDK and client-side support did not make it in time. The proposal is to merge both patches early in Flamingo and then backport to Epoxy.

✅ No concern with microversion usage here.
✅The ordering of microversion support patches across Nova, SDKs, and clients will be handled by respective owners.
✅ Uggla to track that each new microversion in Nova has a corresponding patch in SDK/client layers.
✅ Not directly related to virtiofs, but the new reset-state confirmation prompt in the client was noted and welcomed.

#### One-Time-Use (OTU) Devices ####

OTU devices are designed to be consumed once and then unreserved.
There is a need to provide practical guidance on handling these cleanly, especially in notification-driven environments.

Additionally, there's an important patch related to Placement behavior on over-capacity nodes:
https://review.opendev.org/c/openstack/placement/+/945465
Placement currently blocks new allocations on over-capacity nodes — even if the new allocation reduces usage. This breaks migration from overloaded hosts. The proposed fix allows allocations that do not worsen (or improve) usage.

Note: A similar OTU device handling strategy has been successfully used in Ironic.

✅ Provide an example script or tool for external OTU device cleanup, based on notifications.
✅ Agreement on the proposed Placement fix — it is operator-friendly and resolves real issues in migration workflows.
✅ We likely need to dig deeper into implementation and tooling for broader OTU support.

#### Glance cross-project session ####

Please look at glance summary.

#### Secure RBAC – Finalization Plan ####

Tobias raised concerns about incomplete secure RBAC support in Nova, particularly around default roles and policy behavior. Much of the groundwork has been done, but a number of patches still require review and finalization.

✅ Gmann will continue working on the outstanding patches during the Flamingo cycle. The objective is to complete secure RBAC support in Nova as part of this cycle.

#### Image Properties Handling – DB Schema & API Response ####

The issue arises from discrepancies between image property metadata stored by Nova and what is received from Glance. Nova’s DB schema enforces a 255-character limit on metadata keys and values, which can lead to silent truncation or hard failures (e.g., when prefixing keys like image_ pushes the total length over 255).

Nova stopped supporting custom image properties nearly a decade ago, when the system moved to structured objects (ImageMetaProps via OVO).
Glance still allows some custom metadata, which may be passed through to Nova.
This led to invalid or non-standard keys (e.g., owner_specified.openstack.sha256) being stored or exposed, even though they are not part of Nova’s supported set.

Consensus emerged that we are facing two issues:
- Nova's API may expose more metadata than it should (from Glance).
- Nova stores non-standard or overly long keys/values, resulting in silent truncation or hard DB errors.

✅ Nova should stop storing non-standard image properties altogether.
✅ A cleanup plan should be created to remove existing unused or invalid metadata from Nova's database post-upgrade.
✅ During instance.save(), Nova should identify and delete unused image_* keys from the system metadata table.
✅ We must be cautious to preserve snapshot-related keys that are valid but not part of the base ImageMetaProps.
✅ These changes are considered bugfixes and can proceed without a new spec.

#### Eventlet removal ####

Please read the excellent blog post series from Gibi here: https://gibizer.github.io/posts/Eventlet-Removal-Flamingo-PTG/


#### Enhanced Granularity and Live Application of QoS ####

This was cross team Neutron/Cinder/Nova first topic.

Bloomberg folks presented early ideas around making QoS settings more granular and mutable, and potentially applicable to existing ports or VMs, not just at creation time.

Nova does not operate on multiple instances at once, which conflicts with some proposed behaviors (e.g., live update of QoS on a network/project level).
QoS is currently exposed via flavors in Nova, and is only supported on the frontend for the Libvirt driver.
QoS mutability is non-trivial, with implications for scheduling, resource modeling, and placement interactions.
The scope is broad and would require cross-project collaboration (Neutron, Cinder, Placement).

Use cases and notes from Bloomberg: https://etherpad.opendev.org/p/OpenStack_QoS_Feature_Enhancement_Discussion

✅ Use flavor-based modeling for QoS remains the Nova approach.
✅ Nova should not apply policies across many instances simultaneously.
✅ A spec will be required, especially if new APIs or behavior modifications for existing VMs are introduced. The spec should provide concrete use case examples and API design proposals, including expected behavior during lifecycle operations (resize, rebuild, shelve, etc.).
✅ Max bandwidth adjustments may be possible (as they don’t require reservations), but broader mutability is more complex.
✅ Neutron and Cinder raised no objections regarding Bloomberg’s use cases and proposals. However, please look at Neutron and Cinder's respective summaries.

#### Moving TAP Device Creation from Libvirt to os-vif ####

This change proposes moving the creation of TAP devices from the Libvirt driver into os-vif, making it more consistent and decoupled. However, it introduces upgrade and timing considerations, especially regarding Neutron and OVN behavior.

Bug: https://bugs.launchpad.net/nova/+bug/2073254
Patch: https://review.opendev.org/c/openstack/nova/+/942786


✅ Neutron team is open to adjusting the timing of the "port ready" event, which could eliminate the need for Nova-side hacks.
✅ Sean will proceed with the patch and verify behavior through CI.

#### Instance Annotations, Labels & K8s-Like Semantics ####

Sean proposed introducing a mechanism similar to Kubernetes annotations and labels in Nova, to:
- Express user intent regarding instance behavior (e.g., "should this instance be migrated?")
- Convey lifecycle preferences to external tools like Watcher and Masakari
- Expose capabilities or constraints of an instance (e.g., "cannot be shelved because it has a vTPM")

Proposed Examples of Instance Annotations:
lifecycle:live-migratable=true|false
ha:role=primary|secondary

These would be:
- Set by users (or operators)
- Optionally inherited from flavors (but conflicts would raise 400 Bad Request)
- Expressed intent only — not enforcement of policy

In addition, labels generated by Nova could reflect actual capabilities, like:
lifecycle:live-migratable=false if an instance has a PCI device
lifecycle:shelvable=false if it uses vTPM

✅ Define a new API to expose capabilities of instances (e.g., “can this instance be live-migrated?”)
Values will be derived by Nova based on configuration/hardware and exposed via nova server show.
✅ Sean will create a spec.

✅ Looking at user-defined labels, we eventually considered defining a second API for them to express scheduling/HA preferences.
However we decided the current preferred approach is to start with metadata API, and evolve to a first-class model.
We may need admin-only metadata (e.g., for HA tooling like Masakari) this has been discussed in Admin-Only Instance Metadata / Annotations later point.
✅ Sean will also create a spec for this. (Sean).


#### External Traits and Node Pressure Metrics ####

Sean also proposed allowing external systems (e.g., Watcher, telemetry agents) to annotate compute nodes with traits such as memory/cpu/io pressure, based on /proc/pressure.

Examples:
CUSTOM_MEM_PRESSURE=high
EXTERNAL_IO_PRESSURE=moderate

✅ Support a COMPUTE_MEM_PRESSURE-like trait, populated from sysfs as static info (not dynamic).
✅  A weigher could use these traits to influence placement.Default traits list could be configured (e.g., prefer/avoid hosts with certain pressures or hardware features). This approach could evolve into a generic “preferred traits” weigher, similar to Kubernetes taints/tolerations.

✅Sean will create a dedicated spec for this feature.
✅ Sbauza volunteered to help, especially as the work aligns with weigher logic from the previous cycle.

#### OpenAPI Schema Integration ####

Stephen highlighted that most of the heavy lifting for OpenAPI support is now complete, and the work is down to pure response schema definitions.
This effort spans over three cycles now, and it would be valuable to finalize it early in Flamingo.

✅ We'll formalize this work with a blueprint.
✅ The goal is to make early progress in Flamingo, ideally with a dedicated review day.
✅ Stephen is happy to join synchronous review sessions and will coordinate pings for progress.
✅ Masahito volunteered to help with the remaining work.

#### OpenStack SDK & Client Workflows ####

Stephen raised a few concerns regarding timing mismatches between SDK/OSC freezes and microversion patch merges in Nova.
Some microversion support landed too late to be integrated in the SDK before the Epoxy freeze.

Patches were sometimes missed due to lack of "depends-on" links or broken initial submissions.

✅ Uggla will follow up and finalize these patches early in the Flamingo cycle.

#### Upstream Testing for PCI Passthrough and mdev Devices ####

With IGB support merged in Epoxy, and vIOMMU enabled in some Vexxhost workers (thanks to dansmith), the opportunity exists to expand PCI testing upstream in Tempest.
This would also benefit testing of one-time-use (OTU) devices.

Finalizing mtty testing is a priority, as it helps ensure device support is consistent and regressions (like bug #2098892) are caught early.

✅ Bauzas will lead on wrapping up mtty testing.
✅ Gibi will coordinate with cloud providers to assess Epoxy support and revisit this topic during the next PTG if needed.



#### CPU Power Management – Expected Behavior ####

Melanie raised questions about inconsistencies between design and implementation in Nova’s CPU power management logic. In particular:
- CPUs were being offlined too aggressively, sometimes during reboot or migration operations.
- This contradicts the intent that only unassigned or deallocated cores should be powered off.

There was confusion between two approaches:
- Aggressive power-down of unused CPUs during all idle states (stop, shelve, etc.)
- Conservative behavior, powering off cores only when the VM is deleted or migrated away

Consensus favored the aggressive-but-safe model:
- Power down cores only when not used, e.g., VM is stopped or migrated.
- Be cautious not to power off cores prematurely (e.g., during reboot or verify-resize).

✅ Do not rush to power off CPU cores at compute startup or mid-operation.
✅ Revisit the implementation so the resource tracker runs first, and determines actual core assignments before making decisions.


#### Live Migration with Encrypted Volumes (Barbican Integration) ####

HJ-KIM raised the point that Nova does not currently support live migration of instances using encrypted Cinder volumes managed by Barbican. This is a critical blocker in environments with strict compliance requirements.

✅ This is a parallel issue to vTPM support. We will learn from the vTPM implementation and consider applying similar concepts.
✅ A future solution may involve adjusting how ownership is managed, or providing scoped access via ACLs.
✅ Further discussion/spec work will be needed once an implementation direction is clearer.

#### Manila–Nova Cross-Team Integration ####

The initial Manila–Nova integration is now merged — thanks to everyone involved!

The next step is to:
- Add automated testing (currently manual tests only).
- Start with a few basic positive and negative test scenarios (create, attach, write, delete; snapshot and restore; rule visibility; restricted deletion; etc.).

Additionally, longer-term features and improvements are being considered please look at the etherpad.

✅ We will work on tempest tests.
✅ We will continue enhancing Nova–Manila integration during Flamingo (F) and beyond.
✅ Uggla will submit a spec as needed for land memfd support.


#### Provider Traits Management via provider.yaml ####

📌 Spec: https://review.opendev.org/c/openstack/nova-specs/+/937587
Problem: Traits defined in provider.yaml are added to Placement but never removed if deleted from the file.

✅ Implement a mechanism where Nova copies the applied file to /var/lib/nova/applied_provider.yaml, and diffs it with the active one on restart.
    This would allow traits (and possibly other config) to be safely removed.

   
#### Admin-Only Instance Metadata / Annotations ####

📌 Spec: https://review.opendev.org/c/openstack/nova-specs/+/939190
Issue: Current instance metadata is user-owned, and shouldn't be used by admins.

Proposal: Introduce admin-only annotations (or metadata with ownership tracking), allowing operators to set system-visible metadata without violating user intent.

✅ Introduce a created_by field (similar to locked_by) to track who created metadata: user vs admin.
Consider an admin: prefix namespace for admin-controlled keys (applied to annotations or metadata).
Implementation requires a DB change and a nova-spec.

Note: This aligns well with broader annotation work already discussed in this cycle.


#### delete_on_terminate for Ports (Server Create / Network Attach APIs) ####

📌 Related discussion: https://review.opendev.org/c/openstack/nova-specs/+/936990
Background: This was discussed in past PTGs. Currently, delete_on_terminate can't be updated dynamically across instance lifetime.

✅ A spec with a working PoC will help clarify the desired behavior and unblock the discussion.
Long-term solution may require storing this flag in Neutron as a port property (rather than Nova-specific DB).

#### Graceful Shutdown of Nova Compute Services ####

📌 Spec: https://review.opendev.org/c/openstack/nova-specs/+/937185
Challenge: Need a mechanism to drain compute nodes gracefully before shutdown, without interrupting active workloads or migrations.
 Graceful shutdown is tricky in the presence of live migrations.

Ideas include:
- Temporary “maintenance mode” (block write requests).
- Group-level compute draining.

✅ The topic is important but not urgent — bandwidth is limited.
Note: Eventlet removal may simplify implementing this.
✅ Please report concrete bugs so we understand the blockers.
✅ A nova-spec with PoC would help drive the conversation.

#### Libvirt/QEMU Attributes via Flavor Extra Specs ####

Target: Advanced tuning of I/O performance via iothreads and virtqueue mapping, based on:
https://developers.redhat.com/articles/2024/09/05/scaling-virtio-blk-disk-io-iothread-virtqueue-mapping

✅ Introduce new flavor extra specs such as:
- hw:io_threads=4
- hw:blk_multiqueue=2
These can be added to both flavor and image properties.
✅ A nova-spec should be written to document naming and semantics.


#### Dynamic Modification of libvirt Domain XML (Hook Proposal) ####

oVirt allows for plugins to alter the libvirt domain XML just before instance launch (via VDSM hooks).
Nova does not offer a mechanism to intercept or modify the domain XML, and the design explicitly avoids this.

The desired use case involves injecting configuration that libvirt cannot currently represent, for example, enabling multiuser SPICE consoles.

✅ This proposal is explicitly rejected.
✅ Nova will not support hook points for permuting libvirt XML.
✅ Operators may use out-of-band libvirt/qemu hooks at their own risk, but should not expect upstream support or stability guarantees.


#### Revisiting the "No More API Proxies" Rule ####

Masahito proposed allowing users to filter instances via API based on related service data, such as network_id.

✅ The "no API proxy" rule remains, but with pragmatic exceptions:
- Filtering is acceptable if the data exists in Nova’s DB (e.g., network ID, image ID).
- No cross-service REST calls allowed (e.g., Neutron QoS types still out of scope).
- Filtering by network_id in nova list is reasonable and can proceed.
✅ Masahito will provide a spec.

#### OVN Migration & Port Setup Timing ####

📌 Context: https://bugs.launchpad.net/nova/+bug/2073254
In OVN-based deployments, Neutron signals the network-plugged event too early, before the port is fully set up. This causes issues in live migration, especially under load.

✅ Nova already supports waiting on the network-plugged event. OVN in Ubuntu Noble should behave properly.
A proposal to improve timing in Neutron was discussed (Neutron to wait for port claim in southbound DB).
Nova might support this via a Neutron port hint that triggers tap interface creation earlier during migration (pre-live-migration).
✅ Next step: open an RFE bug in Neutron. If accepted, a Nova spec may be needed.

#### Blocking API Threads During Volume Attachments ####

📌 Context: https://bugs.launchpad.net/nova/+bug/1930406
Volume attachment RPC calls block API workers in uWSGI, leading to starvation when multiple attachments are made in parallel.

✅ Volume/interface attachments should become async, reducing API lock contention.
Fix is non-trivial and will require a microversion.
In the meantime, operators may tune uWSGI workers/threads or serialize attachment calls.

#### Inventory Update Failure – DISK_GB Bug ####

📌 Bug: https://bugs.launchpad.net/nova/+bug/2093869
When local storage becomes temporarily unavailable (e.g., Ceph down), Nova sends total=0 for DISK_GB, which Placement rejects if allocations exist.

✅ The real fix is to restore the storage backend.
Nova should improve error handling/logging, but should not shut down the compute service.

#### Security Group Name Conflict Bug ####

📌 Bug: https://bugs.launchpad.net/nova/+bug/2105896
When multiple security groups share the same name (via Neutron RBAC), instance builds can fail due to incorrect duplicate detection logic.

✅ The issue was fixed in: https://review.opendev.org/c/openstack/nova/+/946079
✅ Fix will be reviewed and backported to Epoxy.

If you've read this far — thank you! 🙏
If you spot any mistakes or missing points, please don't hesitate to let me know.

Best regards.
René.