[Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down
Dear Community, We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment. Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as: [ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level. Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to: Pause or block writes from active VMs when Ceph storage is unavailable Avoid guest OS filesystem corruption Ensure safer recovery when Ceph services are restored Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
Hi Thamanna, Ceph's built to ensure you do not lose any data, so assuming your pools are configured using replicas in threefold, after losing two nodes your PGs should be unavailable. This is by design, and for your own safety. I understand it is not what you're asking for, but my two cents would be to understand why the Ceph nodes hung in the first place and act upon it (fix hardware, maybe upgrade firmware, maybe fix software if applicable). Or in addition extend your Ceph cluster, if there's budget for that. (Always go for dual power supplies and don't even consider non-ECC RAM.) Cheers, Kees On 25/02/2026 05:08, Thamanna Farhath wrote:
Dear Community,
We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment.
Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as:
|[ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0|
It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level.
Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to:
*
Pause or block writes from active VMs when Ceph storage is unavailable
*
Avoid guest OS filesystem corruption
*
Ensure safer recovery when Ceph services are restored
------------------------------------------------------------------------ *Disclaimer :*/ The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.// /
/// /
/E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."// /
its also intentionl that the guest get teh io errors as that is the back presusure/error reporting mechanisium that ceph/linux is intened to provide to userspace application. nova does not and will not monitor the ceph status for you and it wont pause the vms if ceph goes down and that is by design. nova will not take actions on the vm like that unless requested to do so via its api. On 25/02/2026 08:05, Kees Meijs | Nefos wrote:
Hi Thamanna,
Ceph's built to ensure you do not lose any data, so assuming your pools are configured using replicas in threefold, after losing two nodes your PGs should be unavailable. This is by design, and for your own safety.
I understand it is not what you're asking for, but my two cents would be to understand why the Ceph nodes hung in the first place and act upon it (fix hardware, maybe upgrade firmware, maybe fix software if applicable). Or in addition extend your Ceph cluster, if there's budget for that. (Always go for dual power supplies and don't even consider non-ECC RAM.)
Cheers, Kees
On 25/02/2026 05:08, Thamanna Farhath wrote:
Dear Community,
We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment.
Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as:
|[ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0|
It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level.
Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to:
*
Pause or block writes from active VMs when Ceph storage is unavailable
*
Avoid guest OS filesystem corruption
*
Ensure safer recovery when Ceph services are restored
------------------------------------------------------------------------ *Disclaimer :*/The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.// /
/// /
/E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."// /
Dear Community, Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable. However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure. In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption? Thamanna Farhath Associate engineer - R&D 9344093591 mailto:manoj.kumar@zybisys.com thamanna.f@zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ From: Sean Mooney <smooney@redhat.com> To: <openstack-discuss@lists.openstack.org> Date: Wed, 25 Feb 2026 18:20:07 +0530 Subject: Re: [Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down its also intentionl that the guest get teh io errors as that is the back presusure/error reporting mechanisium that ceph/linux is intened to provide to userspace application. nova does not and will not monitor the ceph status for you and it wont pause the vms if ceph goes down and that is by design. nova will not take actions on the vm like that unless requested to do so via its api. On 25/02/2026 08:05, Kees Meijs | Nefos wrote:
Hi Thamanna,
Ceph's built to ensure you do not lose any data, so assuming your pools are configured using replicas in threefold, after losing two nodes your PGs should be unavailable. This is by design, and for your own safety.
I understand it is not what you're asking for, but my two cents would be to understand why the Ceph nodes hung in the first place and act upon it (fix hardware, maybe upgrade firmware, maybe fix software if applicable). Or in addition extend your Ceph cluster, if there's budget for that. (Always go for dual power supplies and don't even consider non-ECC RAM.)
Cheers, Kees
On 25/02/2026 05:08, Thamanna Farhath wrote:
Dear Community,
We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment.
Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as:
|[ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0|
It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level.
Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to:
*
Pause or block writes from active VMs when Ceph storage is unavailable
*
Avoid guest OS filesystem corruption
*
Ensure safer recovery when Ceph services are restored
------------------------------------------------------------------------ *Disclaimer :*/The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.// /
/// /
/E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."// /
Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
Hello Thamanna, The scenario you described, implies your Ceph cluster being that broken it is simply unable to serve any I/O any more. Your virtual machine work load literally experiences the storage being taken away. There is no remedy to cope with that. So, if you're asking about the best practice to recover from such issues: make back-ups (no snapshots, those are not backups) that you periodically test that you can restore. Meanwhile, as mentioned before, I'd suggest to understand why the Ceph nodes were stuck in the first place. Cheers, Kees __ Kees Meijs BICT Nefos Cloud & IT <https://nefos.com/contact> Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931 +31 (0)88 2088 188 <tel:+31882088188> nefos.com <https://nefos.com/contact> The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended. General terms and conditions ("The NLdigital Terms") apply to all our products and services. On 03/03/2026 04:57, Thamanna Farhath wrote:
Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable.
However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure.
In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption?
And one more thought on this: if your Ceph cluster really was inaccessible after a single node failure, then there's a big knowledge gap how to configure Ceph in a professional way. If that is really the case, I recommend to hire a consultant to inspect your cluster setup. Zitat von Kees Meijs | Nefos <keesm@nefos.com>:
Hello Thamanna,
The scenario you described, implies your Ceph cluster being that broken it is simply unable to serve any I/O any more.
Your virtual machine work load literally experiences the storage being taken away. There is no remedy to cope with that.
So, if you're asking about the best practice to recover from such issues: make back-ups (no snapshots, those are not backups) that you periodically test that you can restore.
Meanwhile, as mentioned before, I'd suggest to understand why the Ceph nodes were stuck in the first place.
Cheers, Kees
__
Kees Meijs BICT
Nefos Cloud & IT <https://nefos.com/contact>
Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931
+31 (0)88 2088 188 <tel:+31882088188> nefos.com <https://nefos.com/contact>
The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended.
General terms and conditions ("The NLdigital Terms") apply to all our products and services.
On 03/03/2026 04:57, Thamanna Farhath wrote:
Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable.
However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure.
In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption?
Hello community, Thank you for your explanation and suggestions. Initially, we suspected that the issue was related to Ceph. However, after further testing in our lab environment where both Ceph and OpenStack are deployed together, we observed that the problem appears specifically when a compute node goes down or reboots. In this scenario, the instances that were running on the affected compute node experience persistent I/O errors inside the guest, even after the Ceph cluster regains quorum and becomes healthy again. We also tested instance evacuation to another compute node, but the evacuated instance still encounters the same I/O errors. From our observations, the issue seems to occur only when the compute node failure interrupts ongoing RBD I/O operations, and the instance does not recover properly afterward. Could you please advise on the recommended recovery procedure for such situations? Specifically: Is there a best practice to recover RBD-backed instances after compute node failure when I/O errors appear inside the guest? Are there any recommended Ceph or Nova configurations to prevent this situation or improve recovery? Any guidance would be greatly appreciated. Best regards, Thamanna Farhath From: Eugen Block <eblock@nde.ag> To: <openstack-discuss@lists.openstack.org> Date: Wed, 04 Mar 2026 02:10:12 +0530 Subject: Re: [Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down And one more thought on this: if your Ceph cluster really was inaccessible after a single node failure, then there's a big knowledge gap how to configure Ceph in a professional way. If that is really the case, I recommend to hire a consultant to inspect your cluster setup. Zitat von Kees Meijs | Nefos < mailto:keesm@nefos.com >:
Hello Thamanna,
The scenario you described, implies your Ceph cluster being that broken it is simply unable to serve any I/O any more.
Your virtual machine work load literally experiences the storage being taken away. There is no remedy to cope with that.
So, if you're asking about the best practice to recover from such issues: make back-ups (no snapshots, those are not backups) that you periodically test that you can restore.
Meanwhile, as mentioned before, I'd suggest to understand why the Ceph nodes were stuck in the first place.
Cheers, Kees
__
Kees Meijs BICT
Nefos Cloud & IT < https://nefos.com/contact >
Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931
+31 (0)88 2088 188 <tel:+31882088188> nefos.com < https://nefos.com/contact >
The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended.
General terms and conditions ("The NLdigital Terms") apply to all our products and services.
On 03/03/2026 04:57, Thamanna Farhath wrote:
Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable.
However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure.
In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption?
Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.
E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
Please don’t create new threads with the same topic, it only gets confusing. Have you read this thread? https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... Maybe that’s what you’re seeing? If it is, removing a lock from affected images could help getting rid of the IO errors. I don’t see a way to prevent them though. Compute nodes will fail at some point, Ceph does a great job of keeping data consistent and highly available (if configured correctly). So in case of an unplanned outage you’ll always have to perform some kind of recovery. Zitat von Thamanna Farhath <thamanna.f@zybisys.com>:
Hello community,
Thank you for your explanation and suggestions.
Initially, we suspected that the issue was related to Ceph. However, after further testing in our lab environment where both Ceph and OpenStack are deployed together, we observed that the problem appears specifically when a compute node goes down or reboots.
In this scenario, the instances that were running on the affected compute node experience persistent I/O errors inside the guest, even after the Ceph cluster regains quorum and becomes healthy again.
We also tested instance evacuation to another compute node, but the evacuated instance still encounters the same I/O errors.
From our observations, the issue seems to occur only when the compute node failure interrupts ongoing RBD I/O operations, and the instance does not recover properly afterward.
Could you please advise on the recommended recovery procedure for such situations?
Specifically:
Is there a best practice to recover RBD-backed instances after compute node failure when I/O errors appear inside the guest?
Are there any recommended Ceph or Nova configurations to prevent this situation or improve recovery?
Any guidance would be greatly appreciated.
Best regards,
Thamanna Farhath
From: Eugen Block <eblock@nde.ag> To: <openstack-discuss@lists.openstack.org> Date: Wed, 04 Mar 2026 02:10:12 +0530 Subject: Re: [Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down
And one more thought on this: if your Ceph cluster really was inaccessible after a single node failure, then there's a big knowledge gap how to configure Ceph in a professional way. If that is really the case, I recommend to hire a consultant to inspect your cluster setup.
Zitat von Kees Meijs | Nefos < mailto:keesm@nefos.com >:
Hello Thamanna,
The scenario you described, implies your Ceph cluster being that broken it is simply unable to serve any I/O any more.
Your virtual machine work load literally experiences the storage being taken away. There is no remedy to cope with that.
So, if you're asking about the best practice to recover from such issues: make back-ups (no snapshots, those are not backups) that you periodically test that you can restore.
Meanwhile, as mentioned before, I'd suggest to understand why the Ceph nodes were stuck in the first place.
Cheers, Kees
__
Kees Meijs BICT
Nefos Cloud & IT < https://nefos.com/contact >
Nefos IT bv Burgemeester Mollaan 34a 5582 CK Waalre - NL kvk 66494931
+31 (0)88 2088 188 <tel:+31882088188> nefos.com < https://nefos.com/contact >
The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the organization. No reliance may be placed on this message without written confirmation from an authorised representative of its contents. No guarantee is implied that this message or any attachment is virus free or has not been intercepted and amended.
General terms and conditions ("The NLdigital Terms") apply to all our products and services.
On 03/03/2026 04:57, Thamanna Farhath wrote:
Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable.
However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure.
In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption?
Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.
E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
Dear Community, Thank you for your clarification. We understand that this behavior is by design in Ceph and that OpenStack Nova will not automatically take action when storage becomes unavailable. However, in our case, simply rebooting the affected VMs is not always sufficient. If a crash occurs and persistent I/O errors are seen inside the guest, we would like to understand the recommended recovery procedure. In such scenarios, how can we safely retrieve and restore the instance once Ceph regains quorum? What is the best practice to recover RBD-backed instances after write failures to avoid permanent corruption? From: Sean Mooney <smooney@redhat.com> To: <openstack-discuss@lists.openstack.org> Date: Wed, 25 Feb 2026 18:20:07 +0530 Subject: Re: [Nova][vms]Preventing VM I/O Errors When Ceph OSD Nodes Go Down its also intentionl that the guest get teh io errors as that is the back presusure/error reporting mechanisium that ceph/linux is intened to provide to userspace application. nova does not and will not monitor the ceph status for you and it wont pause the vms if ceph goes down and that is by design. nova will not take actions on the vm like that unless requested to do so via its api. On 25/02/2026 08:05, Kees Meijs | Nefos wrote:
Hi Thamanna,
Ceph's built to ensure you do not lose any data, so assuming your pools are configured using replicas in threefold, after losing two nodes your PGs should be unavailable. This is by design, and for your own safety.
I understand it is not what you're asking for, but my two cents would be to understand why the Ceph nodes hung in the first place and act upon it (fix hardware, maybe upgrade firmware, maybe fix software if applicable). Or in addition extend your Ceph cluster, if there's budget for that. (Always go for dual power supplies and don't even consider non-ECC RAM.)
Cheers, Kees
On 25/02/2026 05:08, Thamanna Farhath wrote:
Dear Community,
We are running OpenStack 2023.1 with Ceph as the backend storage on a 3-node deployment.
Recently, we faced a scenario where two of our servers became unresponsive (hung state), and we had to reboot them. During this time, VMs running on the affected compute node started reporting I/O errors inside the guest OS, such as:
|[ 33.911093] blk_update_request: I/O error, dev vda, sector 229880 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0 [ 33.914953] Buffer I/O error on dev vda1, logical block 319, lost async page write [ 33.914953] Buffer I/O error on dev vda1, logical block 320, lost async page write [ 33.927594] blk_update_request: I/O error, dev vda, sector 229904 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0|
It appears that when Ceph becomes unavailable (or quorum is lost), the VMs continue attempting writes, which results in I/O errors at the guest OS level.
Our goal: We would like to prevent guest filesystem corruption or I/O errors when Ceph is down. Ideally, we want to:
*
Pause or block writes from active VMs when Ceph storage is unavailable
*
Avoid guest OS filesystem corruption
*
Ensure safer recovery when Ceph services are restored
------------------------------------------------------------------------ *Disclaimer :*/The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.// /
/// /
/E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."// /
Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
participants (4)
-
Eugen Block
-
Kees Meijs | Nefos
-
Sean Mooney
-
Thamanna Farhath