- openstack-discuss - lists.openstack.org

[Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down
by Thamanna Farhath 07 Mar '26

07 Mar '26

Dear Community, We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment. Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as: [ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level. Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to: Pause or block writes from active VMs when Ceph storage is unavailable Avoid guest OS filesystem corruption Ensure safer recovery when Ceph services are restored Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

4 8

[Nova][Vms]Persistent I/O Errors in RBD-backed Instances After Compute Node Failure
by Thamanna Farhath 07 Mar '26

07 Mar '26

Hello, We are currently investigating an issue related to OpenStack (2023.1) instances using Ceph RBD storage. However, after further testing in a lab environment we observed that the problem occurs specifically when a compute node goes down or reboots unexpectedly. In this situation, the instances that were running on the affected compute node later experience persistent I/O errors inside the guest OS, even after the compute node comes back online, the instance is usually in a SHUTOFF state, and when we try to start it again, the same I/O errors appear inside the guest. We also attempted instance evacuation to another compute node, but the evacuated instance still shows the same I/O errors inside the guest. We would like to understand the recommended approach in such cases: • What is the best practice to recover RBD-backed instances after a compute node failure when persistent I/O errors appear inside the guest? • Are there any recommended configurations or operational procedures in OpenStack or Ceph to prevent or mitigate this situation? Any suggestions or guidance would be greatly appreciated. Best regards, Thamanna Farhath Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

Re: Update on RDO: A Shift to Containers
by Jose Castro Leon 06 Mar '26

06 Mar '26

We are also heavily affected by this change, as all our deployments at CERN rely on it. Moreover, we are early adopters of RDO and now that we are getting into Epoxy release, it just slips away. The deployment model is not something you can easily change. Does it also mean that our el10 users may never have client libraries packaged to use our setups? And following Massimo remark, will the repo be maintained while it's still in supported status in OpenStack? If not, are you implying that we also need to repackage future security updates? Thanks, Jose CERN Cloud Team ________________________________ From: Massimo Sgaravatto <massimo.sgaravatto(a)gmail.com> Sent: Thursday, March 5, 2026 1:40 PM To: Amy Marrich <amy(a)demarco.com> Cc: RDO Developmen List <dev(a)lists.rdoproject.org>; openstack-discuss <openstack-discuss(a)lists.openstack.org>; users(a)lists.rdoproject.org <users(a)lists.rdoproject.org>; Mike Burns <mburns(a)redhat.com> Subject: Re: Update on RDO: A Shift to Containers A non-negligible impact for our deployment :-( So, as far as I understand, no rpms will be released for the F and G releases, correct? Will you keep releasing updated RPMs for the Epoxy release, while it is in the maintained status, or the repo https://mirror.stream.centos.org/SIGs/9-stream/cloud/x86_64/openstack-epoxy/ will not be updated anymore ? Thanks, Massimo On Wed, Mar 4, 2026 at 4:19 PM Amy Marrich <amy(a)demarco.com<mailto:amy@demarco.com>> wrote: Dear RDO Community/Stakeholders, This email provides an important update regarding the RDO project and its future direction which marks a significant change in how RDO content will be delivered. End of RDO RPM Builds Historically, RDO provided RPM-based rebuilds of upstream OpenStack packages. Due to insufficient contributors, the continuous RDO rpm builds that were released through the CentOS Cloud SIG have been officially discontinued as of the Epoxy release. Introducing the New RDO: Container Focus The future of RDO will be focused entirely on containers. This is a pivot from our traditional RPM delivery, as RDO did not previously focus on containers. RDO will be transitioning to Source-to-Image (S2I) builds for service containers. This work will be starting shortly within the <http://github.com/openstack-k8s-operators> github.com/openstack-k8s-operators<http://github.com/openstack-k8s-operators> organization, and these new container builds will essentially be the "new RDO". Timeline and Usage: * This transition work has not yet begun. The timeline is not committed, but the work is currently expected to be available for the Hibiscus release in late 2026. The final delivery location of those images is still to be determined. * These containers are primarily intended to be consumed via the operators provided in the openstack-k8s-operators GitHub organization. * While nothing explicitly prevents these containers from being used elsewhere, this is not currently part of the plan. Community involvement and feedback for broader usage are welcome including within OKD and the okderators. * Please note that the operators in this organization are currently only developed and tested on Red Hat OpenShift Container Platform. Thank you for your understanding and continued support as we make this important shift. We look forward to engaging with the community on this new path forward. Sincerely, Amy

4 3

[all] Another day another CI failure
by Brian Haley 06 Mar '26

06 Mar '26

Hi, Started seeing tox failures in pep8/docs/unit and some other jobs: AttributeError: 'Parsed' object has no attribute 'config_format' Melanie Witt provided some info at [0][1], I opened a bug [2] as it seems like it was caused by a recent commit. I would hold rechecks until someone confirms it's fixed. -Brian [0] https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&ski… [1] https://paste.opendev.org/show/bS87fFXvqOKODTRMllg4/ [2] https://github.com/tox-dev/tox/issues/3866

3 4

[release] Release countdown for week R-3
by Thierry Carrez 06 Mar '26

06 Mar '26

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, March 12th, 2026. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2026.1 branch will be cut. This branch will track the 2026.1 "Gazpacho" release. Once stable/2026.1 has been created, the master branch will be ready to switch to 2026.2 "Hibiscus" development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2026.1 "Gazpacho" plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2026.1 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2026.1 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2026.2 "Hibiscus") to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2026.1 Gazpacho release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: March 12th, 2026 (R-3 week) Final RC deadline: March 26th, 2026 (R-1 week) Final 2026.1 "Gazpacho" release: April 1st, 2026 2026.2 "Hibiscus" (virtual) PTG: April 20-24, 2026 -- Thierry Carrez (ttx)

1 0

OpenStack RPMs
by Francesco Di Nucci 06 Mar '26

06 Mar '26

Hi all, given that RDO will soon stop releasing RPMs of OpenStack, it has been suggested on IRC etc that AlmaLinux and Rocky Linux communities might be interested in building the RPMs instead, is there already a discussion about it? Best regards Francesco Di Nucci -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

[kolla-ansible][flamingo][horizon] Network creation problem
by Franck VEDEL (UGA) 06 Mar '26

06 Mar '26

Hello everyone, I have a production OpenStack cluster for ESM students running Epoxy, deployed with Kolla-Ansible, and everything is working perfectly. I’m very satisfied with this setup. To prepare for upgrades, I set up a new test cluster in parallel using Kolla-Ansible, but this time with Flamingo. Everything seemed fine until I tried to create a network and its subnet in Horizon. The network is created, but no availability zone is associated with it. As a result, I cannot launch instances on this network. Could this be a change in the new OpenStack installation process that I missed? Or is this a bug? This is the first time I’ve encountered this issue. Is there a parameter that needs to be added in the Horizon override file (_9999-custom-settings.py) to handle this? Thanks in advance for any guidance. Franck VEDEL

2 1

testresources broken with latest fixtures
by Thomas Goirand 06 Mar '26

06 Mar '26

Hi, In its last version of fixtures, the tests folder has been move one directory up. Trying to rebuild testresources, it fails with: ERROR: unittest.loader._FailedTest.testresources.tests.test_test_resource ---------------------------------------------------------------------- ImportError: Failed to import test module: testresources.tests.test_test_resource Traceback (most recent call last): File "/usr/lib/python3.13/unittest/loader.py", line 396, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.13/unittest/loader.py", line 339, in _get_module_from_name __import__(name) ~~~~~~~~~~^^^^^^ File "/build/testresources-nCTnwj/testresources-2.0.2/testresources/tests/test_test_resource.py", line 18, in <module> from fixtures.tests.helpers import LoggingFixture ModuleNotFoundError: No module named 'fixtures.tests' This isn't seen downstream, as it's still possible to run latest versions of both fixtures and testresources, only testresources unit testing is broken with lastest fixtures, but I don't think it creates any issue for the users of these 2 libs. However, on the distro level, I'm not sure how to fix this... :/ Cheers, Thomas Goirand (zigo)

2 1

[puppet][tc] Future of the project
by Takashi Kajinami 06 Mar '26

06 Mar '26

I had similar discussions earlier in #puppet-openstack several times, but I think it's time to start more formal discussions. As some of you may know, Puppet OpenStack project has been facing several challenges for a few years due to the changes made in the external but related communities, and very honestly speaking, the project is at risk. IMO we have two major challenges now. - RDO officially announced that they discontinue RPM builds and focus on new container-base delivery, based on source-to-build. https://lists.rdoproject.org/archives/list/users@lists.rdoproject.org/threa… Because the primary (and only) architecture we currently support is installation from packages, this means CentOS Stream may no longer be in our support scope. CentOS Stream + RDO has been the primary (and only voting) test architecture because it provides the trunk packages which allow us to test the latest code in development. We could technically switch our primary test architecture to the other platforms like Ubuntu or Debian but this means we can't release our modules in coordinated release timings (because we can implement and test the changes for a new version only after the version is released by these distributions) As we've seen some drastic changes triggered by some initiatives (like eventlet removal) and expect more coming soon, this "lagging-behind" development models can cause several problems like [1]. [1] https://review.opendev.org/c/openstack/puppet-neutron/+/978000 - The upstream puppet has been "dead" after Perforce (who acquired puppetlabs) changed their strategies. OpenVox was created as a fork of puppet, but there are number of core modules such as puppetlabs-apache or puppetlabs-mysql which are not yet forked. I've asked possibility to fork and host these in OpenVox but it was not acceptable due to limited resources the OpenVox community and no real commitment (more specifically speaking, sponsorship) I can provide. There are some works done earlier to replace puppet by OpenVox but it is essentially blocked by the fixes needed in these "abandoned" puppetlabs modules. In addtion, we need to bump versions of our testing platform soon, and I expect these unmaintained modules cause new problems. As afar as I'm aware of, migration to CentOS 10 is completely blocked by "unmaintained puppetlabs-mysql[2], and I expect more for migration to Ubuntu 26.04 or Debian 14 we need in upcoming cycles. [2] https://github.com/puppetlabs/puppetlabs-mysql/issues/1676 So... The situation is getting out of my control and I'd admit that I may soon terminate the project in case the situation is not improved, given my work in the OpenStack community has been all voluntary-based these days. I intend to continue my PTL role of this project for next cycle, so that I can help anyone who would be interested in sustaining the project, but I hope that we get some conclusion and agreement about the future project direction and maintenance during this cycle. Please reply to this in thread or reach me directly (in email or irc) if you are interested in further discussions. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

4 5

[neutron] multicast and allowed address pairs
by Bence Romsics 05 Mar '26

05 Mar '26

Hi Neutrinos, We have a user who discovered that two neutron backends behave differently when it comes to handling multicast traffic. This user runs servers that need multicast for VRRP coordination (the point is multicast, not VRRP). They discovered that with the (no longer upstream supported) OpenDaylight backend, they need to let the multicast traffic through with an allowed address pair. On the other hand, the ml2/ovs backend (here always assuming the ovs firewall driver) allows the multicast traffic by default. If you add the same allowed address pair (for example: {'mac_address': '01:00:5e:00:00:64', 'ip_address': '224.0.0.100'}) as with OpenDaylight, it actually breaks multicast traffic. The traffic breaks even on other servers, affected by the same set of flows installed by ovs-agent. My question is: which backend behaves correctly? And do we need to change anything upstream? Since I found the following upstream history... https://opendev.org/openstack/neutron/src/branch/master/doc/source/admin/co… https://bugs.launchpad.net/neutron/+bug/1889631 ... I believe that ml2/ovs (with the ovs fw driver) behaves correctly when it lets the multicast traffic through by default. However I believe it is a bug that the traffic breaks when the user adds the allowed address pair. But I think it would be pretty pointless to start fixing the traffic itself since we would need to make changes for no effect. Rather we could reject allowed address pairs with multicast IPs/MAC addresses, because that would call the attention of the user that multicast traffic is already handled at a different place (that is the default behavior of the fw driver). I also believe that the OpenDaylight driver is buggy when it does not let the multicast traffic through just like ml2/ovs does. However this is not an upstream problem anymore, since networking-odl is long deprecated. Do you agree in the judgement where the bugs are in this case? The possible upstream change: Do you think we should start rejecting allowed address pairs with multicast IPs/MAC addresses? Plus I guess a doc update for allowed address pairs api-ref is due... If needed, I'm happy to bring this to the drivers meeting. It's 100% that I could attend next Friday (6th of March) and 80% for this Friday (27th of February). Thanks, Bence

3 3