Data Center Survival in case of Disaster / HW Failure in DC
List, We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now. The Deployment is as follows. A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode. B. 6 separate Compute nodes C. 3 separate Storage node with Ceph RBD Question is 1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ? 2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs. 3. Is there setup options like a hot standby or similar setups or what we need to employ ? 4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) . 5. Please share your thoughts for reliable crash/fault resistance configuration options in DC. We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO Any thoughts most welcom. Regards, Krish
Hi, first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status. But in case a compute node fails but nova is still responsive a live or cold migration could still be possible to evacuate that host. If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node. So you should have some spare resources to be able to recover from a compute node failure. As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host. You could sustain one node failure without clients noticing it, a second node would cause the cluster to pause. Also you don't have the possibility to recover from a node failure until it is up again, meaning the degraded PGs can't be recovered on a different node. So this also depends on your actual resiliency requirements. If you have a second site you could use rbd mirroring [1] to sync all rbd images between sites. In case the primary site goes down entirely you could switch to the primary site by promoting the rbd images. So you see there is plenty of information to cover and careful planning is required. Regards, Eugen [1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/ Zitat von KK CHN <kkchn.in@gmail.com>:
List,
We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now.
The Deployment is as follows.
A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode.
B. 6 separate Compute nodes
C. 3 separate Storage node with Ceph RBD
Question is
1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ?
2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs.
3. Is there setup options like a hot standby or similar setups or what we need to employ ?
4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) .
5. Please share your thoughts for reliable crash/fault resistance configuration options in DC.
We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO
Any thoughts most welcom.
Regards, Krish
Interesting - we’re starting work on exactly the same analysis at the moment. We’re looking at a separate region for the recovery site, this guarantees no dependencies in the control plane. Ideally, we’d be running active/active for the most critical applications (following AWS recommendations https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-o... <https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html>) but there are some issues we’re working through (such as how to replicate block/object stores between regions). Keeping images/projects in sync between regions also does not seem simple, especially where you want different quotas (e.g. you can have 100 cores in the production site but only 10 by default in the recovery site) As in any DR plan, testing is key - we’ve started to have a look at security groups to do a simulated disconnect test and see what’s not yet in the recovery site. Does anyone have some best practise recommendations or tools for OpenStack disaster recovery ? Cheers Tim
On 5 May 2022, at 14:37, Eugen Block <eblock@nde.ag> wrote:
Hi,
first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status. But in case a compute node fails but nova is still responsive a live or cold migration could still be possible to evacuate that host. If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node. So you should have some spare resources to be able to recover from a compute node failure. As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host. You could sustain one node failure without clients noticing it, a second node would cause the cluster to pause. Also you don't have the possibility to recover from a node failure until it is up again, meaning the degraded PGs can't be recovered on a different node. So this also depends on your actual resiliency requirements. If you have a second site you could use rbd mirroring [1] to sync all rbd images between sites. In case the primary site goes down entirely you could switch to the primary site by promoting the rbd images. So you see there is plenty of information to cover and careful planning is required.
Regards, Eugen
[1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
Zitat von KK CHN <kkchn.in@gmail.com>:
List,
We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now.
The Deployment is as follows.
A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode.
B. 6 separate Compute nodes
C. 3 separate Storage node with Ceph RBD
Question is
1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ?
2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs.
3. Is there setup options like a hot standby or similar setups or what we need to employ ?
4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) .
5. Please share your thoughts for reliable crash/fault resistance configuration options in DC.
We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO
Any thoughts most welcom.
Regards, Krish
This sounds like a great topic we could discuss at the Ops Meetup in Berlin Friday after the Summit. I'm going to plop it in the planning etherpad [1] and we can brainstorm face to face. I encourage any operator of Openstack who's around for the Summit to stick around for an extra day and join us. *nudge Tim* Registration [2] is open. Visit the planning etherpad [1] to propose topics and +1 the ones you are interested it. [1] https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning [2] https://www.eventbrite.com/e/openstack-ops-meetup-tickets-322813472787 -Erik On Thu, May 5, 2022, 10:48 AM Tim Bell <tim.bell@cern.ch> wrote:
Interesting - we’re starting work on exactly the same analysis at the moment.
We’re looking at a separate region for the recovery site, this guarantees no dependencies in the control plane.
Ideally, we’d be running active/active for the most critical applications (following AWS recommendations https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-o...) but there are some issues we’re working through (such as how to replicate block/object stores between regions).
Keeping images/projects in sync between regions also does not seem simple, especially where you want different quotas (e.g. you can have 100 cores in the production site but only 10 by default in the recovery site)
As in any DR plan, testing is key - we’ve started to have a look at security groups to do a simulated disconnect test and see what’s not yet in the recovery site.
Does anyone have some best practise recommendations or tools for OpenStack disaster recovery ?
Cheers Tim
On 5 May 2022, at 14:37, Eugen Block <eblock@nde.ag> wrote:
Hi,
first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status. But in case a compute node fails but nova is still responsive a live or cold migration could still be possible to evacuate that host. If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node. So you should have some spare resources to be able to recover from a compute node failure. As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host. You could sustain one node failure without clients noticing it, a second node would cause the cluster to pause. Also you don't have the possibility to recover from a node failure until it is up again, meaning the degraded PGs can't be recovered on a different node. So this also depends on your actual resiliency requirements. If you have a second site you could use rbd mirroring [1] to sync all rbd images between sites. In case the primary site goes down entirely you could switch to the primary site by promoting the rbd images. So you see there is plenty of information to cover and careful planning is required.
Regards, Eugen
[1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
Zitat von KK CHN <kkchn.in@gmail.com>:
List,
We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now.
The Deployment is as follows.
A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode.
B. 6 separate Compute nodes
C. 3 separate Storage node with Ceph RBD
Question is
1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ?
2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs.
3. Is there setup options like a hot standby or similar setups or what we need to employ ?
4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) .
5. Please share your thoughts for reliable crash/fault resistance configuration options in DC.
We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO
Any thoughts most welcom.
Regards, Krish
Thanks Eugen . I fully agree with not running VMs on Control nodes. When we rolled out the Controller resources we couldn't spare out only as a controller, Becoz of the utilization of the controller host machines resources, so we decided to use them as compute nodes also. On Thu, May 5, 2022 at 6:09 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status.
could you point out the links/docs where I can refer for a proper setup.
But in case a compute node fails but nova is still responsive a live or cold migration could still be possible to evacuate that host.
If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node.
Don't know much about this, any reference is welcome.
So you should have some spare resources to be able to recover from a compute node failure. As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host.
you mean 3 OSD s in single compute node ? I can follow this way is it the best way to do so ?
Any reference to this best ceph deployment model which do the best fault tolerance, kindly share.
You could sustain one node failure without clients noticing it, a second node would cause the cluster to pause. Also you don't have the possibility to recover from a node failure until it is up again, meaning the degraded PGs can't be recovered on a different node. So this also depends on your actual resiliency requirements. If you have a second site you could use rbd mirroring [1] to sync all rbd images between sites.
We have a connectivity link of only 1Gbps between DC and DR and DR is 300 miles away from DC. And the syncing enhancement, how can we achieve this ? Because our HDD writing speeds are too limited, maybe 80 Mbps to 100 Mbps .. SSDs are not available for all compute host machines. Each VM has 800 GB to 1 TB Disk size. Is there any best practice for syncing enhancement mechanisms for HDDs in DC hosts (with a connectivity of 1 Gbps between DR sites with HDD hosts . ?)
In case the primary site goes down entirely you could switch to the primary site by promoting the rbd images. So you see there is plenty of information to cover and careful planning is required.
Regards, Eugen
Thanks again for sharing your thoughts.
[1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
Zitat von KK CHN <kkchn.in@gmail.com>:
List,
We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now.
The Deployment is as follows.
A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode.
B. 6 separate Compute nodes
C. 3 separate Storage node with Ceph RBD
Question is
1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ?
2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs.
3. Is there setup options like a hot standby or similar setups or what we need to employ ?
4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) .
5. Please share your thoughts for reliable crash/fault resistance configuration options in DC.
We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO
Any thoughts most welcom.
Regards, Krish
Hi,
first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status.
could you point out the links/docs where I can refer for a proper setup.
you could check out the basic Openstack HA docs [2] which are outdated but can help to get a better understanding. It also contains a short section about HA compute nodes [3] which basically points to different sources, maybe check out [4], it also covers some basic aspects and refers to some possible solutions. But as I already wrote, that information is old, you'll need to follow up on Masakari or one of the other solutions presented there to see what the current status is. Because the HA docs [2] are not really detailed we had to collect most of the required information ourselves and I wrote some of it down in a blog post [5].
As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host.
you mean 3 OSD s in single compute node ? I can follow this way is it the best way to do so ?
No, I don't talk about compute nodes here but ceph. You only mention that you have three ceph nodes, we don't more so it's difficult to comment.
If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node.
Don't know much about this, any reference is welcome.
I don't have a reference but only my blog post again [6] which contains an example, but I advise against tweaking the database if you don't know what you're doing just because a blog post says so. ;-) There are so many aspects to consider, maybe you should consider professional consulting? Regards, Eugen [2] https://docs.openstack.org/ha-guide/intro-ha.html [3] https://docs.openstack.org/ha-guide/compute-node-ha.html [4] http://aspiers.github.io/openstack-summit-2016-austin-compute-ha/#/agenda [5] https://heiterbiswolkig.blogs.nde.ag/2020/08/21/openstack-ha-upgrade-part1/ [6] https://heiterbiswolkig.blogs.nde.ag/2020/09/25/openstack-ha-upgrade-part-v/ Zitat von KK CHN <kkchn.in@gmail.com>:
Thanks Eugen . I fully agree with not running VMs on Control nodes. When we rolled out the Controller resources we couldn't spare out only as a controller, Becoz of the utilization of the controller host machines resources, so we decided to use them as compute nodes also.
On Thu, May 5, 2022 at 6:09 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
first, I wouldn't run VMs on control nodes, that way you mix roles (control and compute) and in case that one control node fails the VMs are not available. That would not be the case if the control node is only a control node and is also part of a highly available control plane (like yours appears to be). Depending on how your control plane is defined, the failure of one control node should be tolerable. There has been some work on making compute nodes highly available but I don't know the current status.
could you point out the links/docs where I can refer for a proper setup.
But in case a compute node fails but nova is still responsive a live or cold migration could still be possible to evacuate that host.
If a compute node fails and is unresponsive you'll probably need some DB tweaking to revive VMs on a different node.
Don't know much about this, any reference is welcome.
So you should have some spare resources to be able to recover from a compute node failure. As for ceph it should be configured properly to sustain the loss of a defined number of nodes or disks, I don't know your requirements. If your current cluster has "only" 3 nodes you probably run replicated pools with size 3 (I hope) with min_size 2 and failure-domain host.
you mean 3 OSD s in single compute node ? I can follow this way is it the best way to do so ?
Any reference to this best ceph deployment model which do the best fault tolerance, kindly share.
You could sustain one node failure without clients noticing it, a second node would cause the cluster to pause. Also you don't have the possibility to recover from a node failure until it is up again, meaning the degraded PGs can't be recovered on a different node. So this also depends on your actual resiliency requirements. If you have a second site you could use rbd mirroring [1] to sync all rbd images between sites.
We have a connectivity link of only 1Gbps between DC and DR and DR is 300 miles away from DC. And the syncing enhancement, how can we achieve this ? Because our HDD writing speeds are too limited, maybe 80 Mbps to 100 Mbps .. SSDs are not available for all compute host machines. Each VM has 800 GB to 1 TB Disk size.
Is there any best practice for syncing enhancement mechanisms for HDDs in DC hosts (with a connectivity of 1 Gbps between DR sites with HDD hosts . ?)
In case the primary site goes down entirely you could switch to the primary site by promoting the rbd images. So you see there is plenty of information to cover and careful planning is required.
Regards, Eugen
Thanks again for sharing your thoughts.
[1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
Zitat von KK CHN <kkchn.in@gmail.com>:
List,
We are having an old cloud setup with OpenStack Ussuri usng Debian OS, (Qemu KVM ). I know its very old and we can't upgrade to to new versions right now.
The Deployment is as follows.
A. 3 Controller in (cum compute nodes . VMs are running on controllers too..) in HA mode.
B. 6 separate Compute nodes
C. 3 separate Storage node with Ceph RBD
Question is
1. In case of any Sudden Hardware failure of one or more controller node OR Compute node OR Storage Node what will be the immediate redundant recovery setup need to be employed ?
2. In case H/W failure our recovery need to as soon as possible. For example less than30 Minutes after the first failure occurs.
3. Is there setup options like a hot standby or similar setups or what we need to employ ?
4. To meet all RTO (< 30 Minutes down time ) and RPO(from the exact point of crash all applications and data must be consistent) .
5. Please share your thoughts for reliable crash/fault resistance configuration options in DC.
We have a remote DR setup right now in a remote location. Also I would like to know if there is a recommended way to make the remote DR site Automatically up and run ? OR How to automate the service from DR site to meet exact RTO and RPO
Any thoughts most welcom.
Regards, Krish
participants (4)
-
Erik McCormick
-
Eugen Block
-
KK CHN
-
Tim Bell