November 2025 - openstack-discuss

[kolla-ansible] Upgrade from 2024.1 to 2024.1 neutron-l3-agent unhealthy
by hamid.lotfi＠gmail.com 05 Nov '25

05 Nov '25

Hi everyone, I’m hitting a weird issue after a kolla-ansible upgrade and would appreciate any pointers. Environment kolla-ansible 18.8.1 rabbitmq 4.1.3 (deployed by Kolla) upgrade path: 2024.1 → 2025.1 What happened Ran kolla-ansible -i … upgrade from 2024.1 to 2025.1. The upgrade finished without obvious errors in kolla logs. Right after the upgrade every neutron-l3-agent container on all control/compute nodes is marked unhealthy (no ERROR/FATAL lines in the container logs). All tenant instances lose external connectivity. Simply docker restart neutron_l3_agent on each node is enough: containers become healthy and external access is restored instantly. No other OpenStack service shows the same behaviour. Any idea why the agents start in “unhealthy” state until manually restarted? Happy to collect more data if you tell me what to look for.

1 0

[ops][largescale-sig][scientific-sig] Ops Radio Hour on October 31, 2025 at the PTG
by Ildiko Vancsa 04 Nov '25

04 Nov '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __October 31st, 2025 at 1300UTC,__ which is during the PTG! The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. The main topic of the next gathering will be to discuss how to refactor Ops-related activities. See Thierry's email for more details: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… If you have further topics you would like to dive into during the call, please add them to this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org<https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on October 31st! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 1

[horizon] Cancelling weekly meeting tomorrow
by Tatiana Ovchinnikova 04 Nov '25

04 Nov '25

Hello team, I don't have anything to share after the PTG, so I'm cancelling the IRC meeting tomorrow, November 5th. Let's save some time for the reviews and see you next week! With kind regards, Tatiana

1 0

[ironic][all] Making core teams more accessible
by Jay Faulkner 04 Nov '25

04 Nov '25

I'm going to preface this with a comment that I'm not completely sold on this idea yet, but I think it's worth a larger discussion and might be a seed to help with discussion. I've tagged Ironic as this proposal is for Ironic specifically, but I'm very, very interested in the larger community opinion for this kind of approach so I also tagged all. One of the toughest parts about walking the path to becoming core is how fuzzy the targets are. With Ironic recently having added a two-tier review system, with reviewers and approvers separate, we have an opportunity to allow a much, much bigger tent for the ironic-reviewers group without risking code getting merged without approval by one of our folks with more experience and context in OpenStack and Ironic as a whole. So here's the suggestion -- we come up with a static criteria and period with a clear bar to "if you do this, you are an ironic-reviewer". For instance, we could say (I'm not sure about these numbers): if you do an average of Y (5?) reviews/week and have merged a major Ironic feature or multiple bugfix patches, you would automatically be a reviewer. Obviously, we'd have to keep safeguards to prevent abuse; in our community in the past we had a rule that changes required review from two separate corporate entities -- in this case, I'd suggest a waiting period to prevent quick-merging by someone with a single perspective. Exceptions would be, as in the current system, for emergency CI fixes and documentation improvements. To be clear: the safeguards shouldn't be seen as a lack of trust, but maybe instead as an acknowledgement that structural controls hold value even if there's no reason to assume mistrust. Full proposal: * ironic-approvers (+2 and Workflow +1) continues to operate as usual, with admission being based on technical merit, trust, and review frequency. * ironic-reviewers (+2 only) becomes a data-driven group, where reviewers are added/removed based on individual performance metrics TBD (likely reviews/commits/bugs fixed). -- I'll note that today, there's not really any set of reasonable metrics that would add additional reviewers to this group. * Code changes would either need a review from two separate company-representatives, or to have sat for a short waiting period to ensure other folks had a chance to look at it. * Urgent CI fixes, trivial changes, and documentation updates would still only need one review to land. What do folks think? Would this be a good carrot for existing non-cores to up their review and upstream contribution commitments? - Jay Faulkner G-Research OSS

6 6

[nova] Patch for Calico with libvirt >= 9.5.0
by Nell Jerram 04 Nov '25

04 Nov '25

I apologise if this is the wrong place to ask, but is there something I can do to encourage review and approval for https://review.opendev.org/c/openstack/nova/+/960284 ? Many thanks - Nell

2 2

[manila] Cancelling Nov 6th weekly meeting
by Carlos Silva 04 Nov '25

04 Nov '25

Hello Zorillas and interested stackers, As discussed in the previous upstream weekly meeting, we're cancelling this week's meeting, so we have some extra time to go through PTG action items before we regroup. The next meeting will be on Nov 13th. Thank you, carloss

1 0

[kolla-ansible] Migration from OVS to OVN
by Winicius Allan 04 Nov '25

04 Nov '25

Hi kolla o/ I was searching to see if Kolla currently supports migration from OVS to OVN, and I've found this email thread [1]. I'd like to know if this feature has been implemented, and if so, what the necessary steps are. [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032785.h… Best regards

2 2

Kolla-Ansible deployment results in broken dashboard
by Ricardo Matheus 04 Nov '25

04 Nov '25

Hi all I just deployed a multinode OpenStack using Kolla-Ansible, but it seems that the Horizon image (ubuntu / master) is somewhat broken. Horizon runs and the dashboard is accessible, but every file in the "static" part of the web server is not available (no logo images, no css, nothing... just plain unformatted html). Is this a known bug? (btw, there was a guy asking about the same problem in irc a few days ago, it is in the logs for 2025-10-23 (Thursday) <https://meetings.opendev.org/irclogs/%23openstack/%23openstack.2025-10-23.l…> ) Cheers! RDM

2 2

[ptg] #eventlet-removal 2026.1 Gazpacho PTG summary
by Herve Beraud 04 Nov '25

04 Nov '25

Hi everyone, Following our last week PTG meeting, I’d like to share a brief summary of the key outcomes, achievements, and next steps we agreed upon for the Eventlet removal initiative. ## Flamingo Retrospective We have crossed an important milestone: over 50% of impacted projects have shown significant migration progress. 19 deliverables are fully migrated, while 32 others are currently undergoing migration work. Here are the main accomplishments delivered during Flamingo: - Eventlet now includes support for Python 3.14, ensuring compatibility with upcoming Python releases; - Oslo.service introduced a new threading backend to provide a fully threading-based execution model; - Ironic, Heat, Barbican, Mistral, and several other projects are now fully migrated; - the Neutron migration is just one step away from completion; - Nova-api, nova-metadata-api, nova-conductor nova-scheduler can now run in threading mode. ## Gazpacho Call to Actions The following actions are expected to structure our efforts for the Gazpacho development cycle: - we must enhance oslo.service to prevent fork-inherited lock deadlocks by defaulting to the spawn multiprocessing start method, which will address known Manila and Cinder bugs; WIP https://review.opendev.org/c/openstack/oslo.service/+/963742 - we are preparing to merge the new opt-in H905 hacking rule to ensure consistent Eventlet usage detection across all OpenStack projects (see: https://review.opendev.org/c/openstack/hacking/+/959699) - we should clean up the bug tracker of migrated projects to reflect the current status of the eventlet removal efforts. Many of those projects surely have historical eventlet bugs that could be closed; - we need to maintain momentum on ongoing migrations and accelerate progress on the most delayed projects; - we should prioritize the “low-hanging-fruit” migrations, such as Cyborg and Magnum. ## Project Needing Attention Several projects have not shown any notable update during this cycle and require focus in the next series. These projects still contain Eventlet occurrences that must be addressed urgently: Blazar, Venus, Tacker, Masakari, and Zun. The TC has marked Venus and Vitrage as inactive, and these teams may be retired in 2026.2 if activity does not resume. ## Open Discussion Nova plans to focus on nova-compute, nova-novncproxy, testing improvements, and Rally-based benchmarking to fine-tune thread pool defaults, making Gazpacho the likely cycle where Nova fully switches from Eventlet to threading. Dan Smith presented a strategy idea for Glance that we could name the “Optional Eventlet Shim Strategy.” The goal of this approach is to make the use of Eventlet fully optional, allowing Glance to run without this dependency by default. To achieve this, all Eventlet-related code would be isolated outside of Glance’s core and moved into a separate module acting as a bridge. The core of Glance would remain unique and free of any references to Eventlet. If users still wish to rely on Eventlet, they would be able to install an additional module that provides the required implementation. Without this module, Glance would naturally run using the native threads concurrency mode. This approach offers a smooth transition path while preserving the ability to revert if needed, ultimately moving Glance toward a fully Eventlet-free architecture. Thank you to everyone who contributed during the PTG discussions and for the continuous efforts across teams. Let’s keep the momentum going as we enter the Gazpacho cycle and move closer to a fully Eventlet-free OpenStack. As always, you’re welcome to join us on IRC at #openstack-eventlet-removal, and you can find guidance, resources, and updates at https://removal.eventlet.org/ Thanks for your attention. -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[watcher] 2026.1 Gazpacho PTG summary
by Douglas Viroel 04 Nov '25

04 Nov '25

Hi all, Last week's PTG had a lot of great discussions, covering many different aspects of the Watcher project. If you want to get mode details about a topic, you can find the PTG etherpad with all notes here https://etherpad.opendev.org/p/watcher-2026.1-ptg Here is a summary of the discussions that we had: Future of datasources backends and untested integrations *Discussion*: The discussion revolved around the removal timeline for Monasca, the status of Gnocchi, and the deprecation of Prometheus datasource in favor of Aetos. There was also a proposal to deprecate MAAS support due to lack of testing and documentation, and to re-evaluate experimental integrations in 2026.2. *Agreed*: - AI(jgilaber): Monasca datasource will be removed this cycle. - More research is needed on current Gnocchi usage, and there will be no change in its support this cycle. Suggestion: include a question about gnocchi in the next openstack user survey. - AI(jgilaber): Prometheus will be deprecated in 2026.1 in favor of Aetos. Suggestion: include documentation on how to upgrade between datasources. - AI(dviroel): MAAS will be marked as deprecated this cycle, since it has Eventlet dependent code that is planned to be removed in a near future. We will send another email to the ML to call for maintainers. - AI(sean-k-mooney): Update Watcher documentation about service integrations that are now deprecated. Openstack SDK *Discussion*: It was focused on integrating Watcher with the OpenStack SDK to deprecate and remove the python-bindings in other places. The goal for watcher is to replace the current usage of the project's client with the SDK instead (if there is support for it). The goal for python-watcherclient is to only provide the *openstackclient* plugin and replace python binding by adopting the usage of the SDK (when available). And for the watcher-dashboard is to use the SDK exclusively for API interactions (also when available). *Agreed*: - Start with watcher integrations. The goal is to move one service client (e.g.: Nova) to the SDK this cycle. - Do not freeze python bindings until SDK support is in place. - A single spec can address the overall plan. Code modernization, dependencies and dead code removal *Discussion*: The discussion covered code modernization using pyupdate and potentially ruff for cleanup. It also addressed the removal of dead code, such as API routes that always raised exceptions and commented-out code in production files. The removal of various client dependencies (Neutron, Glance, Monasca) was also proposed. *Agreed*: - Apply the same pre-commit and ruff linting checks to the tempest plugin and watcher client. - Defer decisions on more typing until necessary, possibly starting with interfaces. - Explore dropping the number of dependencies, such as multiple timezone libraries. Applier's Workflow Execution and Its Interface/Contract *Discussion*: The workflow for the Applier was identified as poorly documented, leading to questions during reviews of new actions. The need for a default Action interface was discussed. *Agreed*: - The assessment of a default Action interface will be based on the chosen path for rollback and aborting topics. - AI(dviroel): Current interface needs to be documented regardless. Applier: Aborting running tasks *Discussion*: The current implementation of the Applier spawning a new green thread for each action, and killing threads for actions that support abort(), was discussed. *Agreed*: - Stop spawning/killing threads on every action. This code can be refactored right after we merge the eventlet changes in the applier (so we don't mix the proposals). - Improve the execute() method in actions to check resource status and abort the process/looping when an action is cancelled/aborted. Applier: Rollback of Action Plans *Discussion*: The current lack of a working rollback mechanism and the fact that the revert() method from Actions is not being tested or called were highlighted. The future of the rollback option was debated. New rollback mechanisms, such as a user-triggered "rollback" action for failed action plans, were also considered. *Agreed*: - Auto-revert does not work and should be removed in the future. - AI(dviroel): Current behavior should be treated as a bug, and the documentation and associated configuration options should be updated accordingly. - A new spec should be proposed for a new action plan to revert workflow. CI Testing and Coverage *Discussion*: The naming and refactoring of watcher CI jobs were discussed. The need for every voting job in check to also run in the gate was emphasized, as well as in stable branches and other watcher projects. Job renames and consolidation were proposed, along with enabling tempest scenario jobs for stable branches and creating a new grenade job for upgrade testing. *Agreed*: - AI(dviroel): Job renames/consolidation: watcher-functional can be merged into other tempest jobs, watcher-tempest-actuator can be merged into strategies job, and ipv6 job should be integrated into other existing tempest jobs. Gate updates will occur after renaming and merging jobs. Backporting changes from master to stable branches is expected. - AI(dviroel): In watcher-tempest-plugin: replace tempest-functional with a tempest job that runs scenario tests for stable branch validation. - AI(chandankumar): A new grenade job to test upgrades between slurp releases and include more testing in existing jobs. - AI(sean-k-mooney): Propose a watcher job to run against OpenStack requirements project. Improving testing coverage for strategies by doing functional testing *Discussion*: A specification proposal and a detailed implementation plan for improving testing coverage for strategies through functional testing were presented. *Agreed*: - A phased approach will be taken: - AI(amoralej): 1st phase: API only GETs/POSTs. - 2nd phase: Adding decision-engine + Nova + Prometheus datastore. - 3rd phase: Adding Applier. Rally Testing in watcher *Discussion*: The current status of Rally testing in Watcher was reviewed, noting that the rally-task-watcher job runs but is not in the Watcher repository. Missing functionalities were identified, such as the inability to pass audit template scope and parameters, lack of auto-triggering, and no support for event, continuous audit, action plans, and actions. *Agreed*: - AI(chandankumar): Move rally-openstack watcher plugin code to watcher repo - AI(chandankumar): Add periodic job to run rally jobs - Revisit this topic in future as we progress on scaling watcher CI Watcher-dashboard improvements *Discussion*: Improvements to the Watcher dashboard were discussed, including adding auto-page refresh for audit/action plan status, a start button for action plan details, and options for bulk archiving. Dashboard testing using the Django test framework and Playwright was also covered. *Agreed*: - Create wishlist bugs or blueprints to track dashboard improvements. - AI(chandankumar): Check with TC regarding the usage of pytest to improve wording and proceed with integration tests implementation - AI(reviewers): Get the spec https://review.opendev.org/c/openstack/watcher-specs/+/963438 merged. - A spec will be required for the audit and action plan bulk archive feature. The future of datamodel list API *Discussion*: The utility of the datamodel list API, beyond tempest tests, was questioned. The possibility of freezing the API with existing content to avoid microversion bumps for new instance/node updates was raised. The addition of new storage models (storage and baremetal) was also discussed. *Agreed*: - Do NOT extend the API to support additional models. - Defer the removal of datamodel list (compute model) to future discussions, but for now, do not extend it further, even if new fields are added to model elements. - AI(dviroel): Add test to avoid new API changes in datamodel list. Eventlet Removal *Discussion*: Changes made in the Flamingo cycle regarding Eventlet removal were reviewed.The need for a collector sync timeout for threading mode and refactoring the action plan cancel workflow were discussed. *Agreed*: - AI(dviroel): Collector timeout topic: REST API calls should continue to have their own timeouts, and an event trigger can assist in stopping the overall sync process. Add a new config option for collector timeout should be considered. - Applier: stop killing threads when an action plan is canceled. We can keep the current behavior for evenlet but it will be a noop for threading mode (minimal impact for eventlet removal changes). - AI(dviroel): Set MAAS to deprecate after PTG. We will send an email to the ML to call for maintainers once again. (same AI in Future of datasources topic) Future of Noisy Neighbort Goal/Strategy *Discussion*: The deprecation of cache monitoring metrics in the kernel and Nova, which formed the basis of the current noisy neighbor strategy, was discussed. The need to replace this strategy and identify new metrics for contention (e.g., CPU steal, CPU pressure, IOWait) and noisy neighbors (e.g., CPU usage from low-priority instances) was explored. Instance priority mechanisms were also considered. *Agreed*: - Remove the current noisy neighbor strategy in 2026.2+ to allow deprecation to ship in a Slurp release. - AI(dviroel): a proof of concept for CPU steal/IOWait/other metrics is a nice to have, to replace current LLC monitoring metrics. - Use instance metadata for PoC, but consider different solutions for identifying/classifying workloads priorities, like tiering based on flavor extra spec information. Scaling Watcher *Discussion*: The limitations of running a single instance of the Decision Engine and Applier were discussed, with ideas for horizontal scalability. Scalability concerns were raised for different watcher resources: audits, actions and action plans. Some of the open issues that were discussed: - Centralized datamodel: it is independently managed by each decision-engine. Move the model from in-memory to memcached or the database was discussed. - ONESHOT audits are not a failover in a multiple-decision engine deployment. - When an applier dies, the action plan remains ONGOING and is only cancelled upon the applier restart. The main issue was identified as continuous audits being associated with a specific decision-engine. The concept of moving to an event-driven model with stateless decision engines and appliers was proposed, where data models would reside in the database or a shared data store. *Agreed*: - Fow now, instrument the decision-engine to measure the size of the model, and time taken to process notifications. - Keep current failover behavior for ONESHOT audits. - We could have a service monitor in the applier to reschedule PENDING action plans and cancel ONGOING ones, providing the proper status message. - There should be a way limit and set concurrency of actions in audits. In this cycle we will propose a mechanism setting them at system or audit level that covers all the strategies. Stacking Strategies *Discussion*: The possibility of having stacking strategies was brainstormed in this topic. This included ideas like sequential execution of multiple strategies, where a mutable cluster data model would be shared among them. This could result in a list of linked action plans, or result in merging all actions into a single action plan to avoid unnecessary steps. This discussion may be revisited soon, with a more detailed use case. *Agreed*: - Get back to this topic when we have more detailed use cases. Propose a spec to highlight the need of this feature in watcher. Please let me know if I missed something. Thanks! Assisted-By: Gemini -- Douglas Viroel - dviroel

1 0