- openstack-discuss - lists.openstack.org

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2026.1/R-3)
by Goutham Pacha Ravi 09 Mar '26

09 Mar '26

Hello Stackers, We're three weeks away from the date of the coordinated release of OpenStack 2026.1 "Gazpacho" [1]. At the end of this week, all deliverables following the "cycle-with-rc" release model [2] will have published a release candidate. This should assist the community in testing, finding, and patching any release-critical issues for a timely release on April 01, 2026. Looking ahead into the 2026.2 "Hibiscus" release cycle, the OpenStack Technical Committee has updated testing runtimes [3]. We are strongly recommending testing all Python deliverables shipping in the Hibiscus release with Python 3.14; although the common CI template for unit testing will run the job as "non-voting". When we have an operating system available that ships Python 3.14 natively (such as the impending Ubuntu 26.04 LTS), test jobs are expected to become smoother and require less time and resources. PTL Elections are ongoing for Horizon and Barbican [4]. If you are an Active Project Contributor (and an OpenInfra Foundation Individual Member) who hasn't voted yet, please do exercise your vote. If you're having trouble finding your ballot, please reach out to election officials via this email list or ping them within #openstack-election on OFTC. I'd like to congratulate our latest Outreachy Intern, Rose Kimondo, for successfully completing her internship with the OpenStack Manila and Horizon project teams [5]. Rose worked on significant improvements to Manila UI, and Carlos Eduardo da Silva (carloss) and I had the pleasure of mentoring her. If you are aware of opportunities that would retain Rose in our community, please reach out directly to her. === Weekly Meeting === The TC has officially re-formed for the 2026.2 cycle, and our last weekly IRC meeting was held on March 03, 2026 [6]. Our immediate focus was establishing a steady administrative foundation. We'll be electing a new Chair, refreshing our liaisons for the Vulnerability Management, Election and DPL teams, and auditing our weekly meeting time to account for upcoming Daylight Saving Time changes in some parts of the world. Beyond these logistics, we discussed ongoing CI breakages in the requirements repository related to setuptools pinning and a particularly thorny issue where an abandoned dependency is currently blocking Horizon translations [7]. We also discussed "leaderless" project teams. For projects like Masakari, which shows promise in the current market conditions (such as VMware-to-OpenStack migrations), we are exploring a transition to the Distributed Project Leadership (DPL) model to spread the governance load among active contributors. Projects like Vitrage and Venus have seen prolonged inactivity without clear revival interest; we are moving toward their formal retirement. This serves our broader objective of directing community energy toward projects that demonstrate both active maintenance and clear future utility, rather than maintaining technical debt. The next meeting of the OpenStack Technical Committee is today, March 10, 2026, at 1700 UTC. This meeting will be hosted on the #openstack-tc channel on OFTC. Please see the agenda on the meeting's wiki page [8]. I hope to see you there. === Upcoming Events === - 2026-03-10: OpenInfra Board Meeting: https://board.openinfra.org/ - 2026-03-12: 2026.1 "Gazpacho" Release Candidates due - 2026-03-23: KubeCon+CloudNativeCon EU begins: https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/ - 2026-04-20: OpenInfra PTG begins: https://openinfra.org/ptg/ - 2026-04-25: OpenInfra Days, Nigeria: https://www.meetup.com/openstack-nigeria-user-group/events/313525373/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) [1] 2026.1 "Gazpacho" Release Schedule: https://releases.openstack.org/gazpacho/schedule.html [2] OpenStack release models: https://releases.openstack.org/reference/release_models.html [3] Tested runtimes for OpenStack 2026.2: https://governance.openstack.org/tc/reference/runtimes/2026.2.html [4] OpenStack 2026.2 PTL Elections: https://governance.openstack.org/election/ [5] Rose Kimondo's Outreachy Internship blog: https://rosekimondo.dev/blog/ [6] TC Meeting IRC Log, March 03 2026: https://meetings.opendev.org/meetings/tc/2026/tc.2026-03-03-17.00.html [7] Horizon translations blocked: https://github.com/Zegocover/enmerkar/pull/8 [8] TC Meeting Agenda, March 10 2026: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Re: [Nova][Vms]Persistent I/O Errors in RBD-backed Instances After Compute Node Failure
by Thamanna Farhath 09 Mar '26

09 Mar '26

Hi Eugen, Thank you for the guidance. The solution worked and the instance was able to start successfully again after removing the stale lock and blocklisting the old client. I had one follow-up question regarding operations in larger environments. In our production setup we run 100+ VMs, and handling these situations manually (checking RBD status, blocklisting clients, and removing locks) would be difficult if multiple hosts or instances are affected. Is there any OpenStack service or recommended approach that automatically handles stale RBD locks when a compute host goes down or during evacuation? For example, does Nova/libvirt handle lock cleanup during recovery, or are there any Ceph/OpenStack best practices or automation mechanisms to avoid manual intervention in such cases? Thanks again for your help. Best regards, Thamanna From: Kevin Le coq <kevin.lecoq(a)protonmail.ch> To: "Thamanna Farhath"<thamanna.f(a)zybisys.com> Cc: "openstack-discuss"<openstack-discuss(a)lists.openstack.org> Date: Sat, 07 Mar 2026 21:47:04 +0530 Subject: Re: [Nova][Vms]Persistent I/O Errors in RBD-backed Instances After Compute Node Failure Hello, From experience, this type of behavior can sometimes be related to a Ceph keyring configuration issue. In particular, if the Ceph client used by Nova does not have the appropriate capabilities, Nova may fail to properly reacquire the lock on the RBD image after the compute node goes down or reboots unexpectedly. In such cases, even after the compute node returns or the instance is evacuated to another node, the instance may continue to experience persistent I/O errors inside the guest. Could you please verify that the Ceph keyring used by Nova has the correct RBD profile enabled for the pool used by your instances? For example: [client.openstack] key = "<redacted>" caps mgr = allow * caps mon = profile rbd caps osd = profile rbd pool=vms Ensuring that the client has the proper profile rbd permissions on the relevant pool allows Nova to correctly manage and recover RBD locks when instances are restarted or moved to another compute node. It may also be helpful to check for stale locks on the affected RBD images and verify that the compute nodes are using the same Ceph client configuration. Best regards, Kevin Le coq Le samedi 7 mars 2026 à 15:49, Thamanna Farhath < mailto:thamanna.f@zybisys.com > a écrit : Hello, We are currently investigating an issue related to OpenStack (2023.1) instances using Ceph RBD storage. However, after further testing in a lab environment we observed that the problem occurs specifically when a compute node goes down or reboots unexpectedly. In this situation, the instances that were running on the affected compute node later experience persistent I/O errors inside the guest OS, even after the compute node comes back online, the instance is usually in a SHUTOFF state, and when we try to start it again, the same I/O errors appear inside the guest. We also attempted instance evacuation to another compute node, but the evacuated instance still shows the same I/O errors inside the guest. We would like to understand the recommended approach in such cases: • What is the best practice to recover RBD-backed instances after a compute node failure when persistent I/O errors appear inside the guest? • Are there any recommended configurations or operational procedures in OpenStack or Ceph to prevent or mitigate this situation? Any suggestions or guidance would be greatly appreciated. Best regards, Thamanna Farhath Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email." Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

[kolla-ansible][flamingo][vpnaas] Problem and solution
by Franck VEDEL (UGA) 09 Mar '26

09 Mar '26

1 0

[neutron] Bug deputy - week of March 2nd
by Slawek Kaplonski 09 Mar '26

09 Mar '26

Hi, I was bug deputy last week. It was fairly quiet week regarding neutron bugs (there were issues impacting our CI but those were not really Neutron related). ### Critical - https://bugs.launchpad.net/neutron/+bug/2143144 - [networking-sfc] py312 job broken - fixed already by Rodolfo with: https://review.opendev.org/c/openstack/networking-sfc/+/978762 ### High - https://bugs.launchpad.net/neutron/+bug/2143336 - [FT] test_gateway_chassis_least_loaded_scheduler_anti_affinity unstable - **unassigned**, gate failure issue, for now we are skipping this unstable test ### Low - https://bugs.launchpad.net/neutron/+bug/2143050 - Auto allocate service plugin should inherit from ``neutron_lib.services.base.ServicePluginBase`` - fix proposed by Rodolfo: https://review.opendev.org/c/openstack/neutron/+/978898, - https://bugs.launchpad.net/neutron/+bug/2143233 - # [neutron-fwaas] Document: OVN driver has no ``logapi`` implementation - documentation improvement, I assigned it to myself, - https://bugs.launchpad.net/neutron/+bug/2143560 - # Optimization: create a bulk operation for ``_create_port_security_group_binding`` - unassigned, seems a bit like a low-hanging-fruit -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[puppet][tc] Future of the project
by Takashi Kajinami 09 Mar '26

09 Mar '26

I had similar discussions earlier in #puppet-openstack several times, but I think it's time to start more formal discussions. As some of you may know, Puppet OpenStack project has been facing several challenges for a few years due to the changes made in the external but related communities, and very honestly speaking, the project is at risk. IMO we have two major challenges now. - RDO officially announced that they discontinue RPM builds and focus on new container-base delivery, based on source-to-build. https://lists.rdoproject.org/archives/list/users@lists.rdoproject.org/threa… Because the primary (and only) architecture we currently support is installation from packages, this means CentOS Stream may no longer be in our support scope. CentOS Stream + RDO has been the primary (and only voting) test architecture because it provides the trunk packages which allow us to test the latest code in development. We could technically switch our primary test architecture to the other platforms like Ubuntu or Debian but this means we can't release our modules in coordinated release timings (because we can implement and test the changes for a new version only after the version is released by these distributions) As we've seen some drastic changes triggered by some initiatives (like eventlet removal) and expect more coming soon, this "lagging-behind" development models can cause several problems like [1]. [1] https://review.opendev.org/c/openstack/puppet-neutron/+/978000 - The upstream puppet has been "dead" after Perforce (who acquired puppetlabs) changed their strategies. OpenVox was created as a fork of puppet, but there are number of core modules such as puppetlabs-apache or puppetlabs-mysql which are not yet forked. I've asked possibility to fork and host these in OpenVox but it was not acceptable due to limited resources the OpenVox community and no real commitment (more specifically speaking, sponsorship) I can provide. There are some works done earlier to replace puppet by OpenVox but it is essentially blocked by the fixes needed in these "abandoned" puppetlabs modules. In addtion, we need to bump versions of our testing platform soon, and I expect these unmaintained modules cause new problems. As afar as I'm aware of, migration to CentOS 10 is completely blocked by "unmaintained puppetlabs-mysql[2], and I expect more for migration to Ubuntu 26.04 or Debian 14 we need in upcoming cycles. [2] https://github.com/puppetlabs/puppetlabs-mysql/issues/1676 So... The situation is getting out of my control and I'd admit that I may soon terminate the project in case the situation is not improved, given my work in the OpenStack community has been all voluntary-based these days. I intend to continue my PTL role of this project for next cycle, so that I can help anyone who would be interested in sustaining the project, but I hope that we get some conclusion and agreement about the future project direction and maintenance during this cycle. Please reply to this in thread or reach me directly (in email or irc) if you are interested in further discussions. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

4 6

[keystone] App Credential reactivation after reconnection is not immediate
by Vincent Godin 08 Mar '26

08 Mar '26

If you create a 24-hour credential application with a non-local user who has a 2-hour time-to-live (default_authorization_ttl), after 2 hours the TTL resets to 0 and the credential application stops working, which is expected. If the user logs back in, there's a waiting period before the credential application becomes functional again, and the same applies if the user want to create a new credential application. I've noticed that if the user leaves a group with member privileges, for example, their app credentials are immediately invalidated. The same behavior should occur when the user logs back in after the TTL expires.

2 1

[Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down
by Thamanna Farhath 07 Mar '26

07 Mar '26

Dear Community, We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment. Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as: [ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level. Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to: Pause or block writes from active VMs when Ceph storage is unavailable Avoid guest OS filesystem corruption Ensure safer recovery when Ceph services are restored Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

4 8

[Nova][Vms]Persistent I/O Errors in RBD-backed Instances After Compute Node Failure
by Thamanna Farhath 07 Mar '26

07 Mar '26

Hello, We are currently investigating an issue related to OpenStack (2023.1) instances using Ceph RBD storage. However, after further testing in a lab environment we observed that the problem occurs specifically when a compute node goes down or reboots unexpectedly. In this situation, the instances that were running on the affected compute node later experience persistent I/O errors inside the guest OS, even after the compute node comes back online, the instance is usually in a SHUTOFF state, and when we try to start it again, the same I/O errors appear inside the guest. We also attempted instance evacuation to another compute node, but the evacuated instance still shows the same I/O errors inside the guest. We would like to understand the recommended approach in such cases: • What is the best practice to recover RBD-backed instances after a compute node failure when persistent I/O errors appear inside the guest? • Are there any recommended configurations or operational procedures in OpenStack or Ceph to prevent or mitigate this situation? Any suggestions or guidance would be greatly appreciated. Best regards, Thamanna Farhath Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Re: Update on RDO: A Shift to Containers
by Jose Castro Leon 06 Mar '26

06 Mar '26

We are also heavily affected by this change, as all our deployments at CERN rely on it. Moreover, we are early adopters of RDO and now that we are getting into Epoxy release, it just slips away. The deployment model is not something you can easily change. Does it also mean that our el10 users may never have client libraries packaged to use our setups? And following Massimo remark, will the repo be maintained while it's still in supported status in OpenStack? If not, are you implying that we also need to repackage future security updates? Thanks, Jose CERN Cloud Team ________________________________ From: Massimo Sgaravatto <massimo.sgaravatto(a)gmail.com> Sent: Thursday, March 5, 2026 1:40 PM To: Amy Marrich <amy(a)demarco.com> Cc: RDO Developmen List <dev(a)lists.rdoproject.org>; openstack-discuss <openstack-discuss(a)lists.openstack.org>; users(a)lists.rdoproject.org <users(a)lists.rdoproject.org>; Mike Burns <mburns(a)redhat.com> Subject: Re: Update on RDO: A Shift to Containers A non-negligible impact for our deployment :-( So, as far as I understand, no rpms will be released for the F and G releases, correct? Will you keep releasing updated RPMs for the Epoxy release, while it is in the maintained status, or the repo https://mirror.stream.centos.org/SIGs/9-stream/cloud/x86_64/openstack-epoxy/ will not be updated anymore ? Thanks, Massimo On Wed, Mar 4, 2026 at 4:19 PM Amy Marrich <amy(a)demarco.com<mailto:amy@demarco.com>> wrote: Dear RDO Community/Stakeholders, This email provides an important update regarding the RDO project and its future direction which marks a significant change in how RDO content will be delivered. End of RDO RPM Builds Historically, RDO provided RPM-based rebuilds of upstream OpenStack packages. Due to insufficient contributors, the continuous RDO rpm builds that were released through the CentOS Cloud SIG have been officially discontinued as of the Epoxy release. Introducing the New RDO: Container Focus The future of RDO will be focused entirely on containers. This is a pivot from our traditional RPM delivery, as RDO did not previously focus on containers. RDO will be transitioning to Source-to-Image (S2I) builds for service containers. This work will be starting shortly within the <http://github.com/openstack-k8s-operators> github.com/openstack-k8s-operators<http://github.com/openstack-k8s-operators> organization, and these new container builds will essentially be the "new RDO". Timeline and Usage: * This transition work has not yet begun. The timeline is not committed, but the work is currently expected to be available for the Hibiscus release in late 2026. The final delivery location of those images is still to be determined. * These containers are primarily intended to be consumed via the operators provided in the openstack-k8s-operators GitHub organization. * While nothing explicitly prevents these containers from being used elsewhere, this is not currently part of the plan. Community involvement and feedback for broader usage are welcome including within OKD and the okderators. * Please note that the operators in this organization are currently only developed and tested on Red Hat OpenShift Container Platform. Thank you for your understanding and continued support as we make this important shift. We look forward to engaging with the community on this new path forward. Sincerely, Amy

4 3

[all] Another day another CI failure
by Brian Haley 06 Mar '26

06 Mar '26

Hi, Started seeing tox failures in pep8/docs/unit and some other jobs: AttributeError: 'Parsed' object has no attribute 'config_format' Melanie Witt provided some info at [0][1], I opened a bug [2] as it seems like it was caused by a recent commit. I would hold rechecks until someone confirms it's fixed. -Brian [0] https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&ski… [1] https://paste.opendev.org/show/bS87fFXvqOKODTRMllg4/ [2] https://github.com/tox-dev/tox/issues/3866

3 4