Open Stack

Hi,

>> first, I wouldn't run VMs on control nodes, that way you mix roles
>> (control and compute) and in case that one control node fails the VMs
>> are not available. That would not be the case if the control node is
>> only a control node and is also part of a highly available control
>> plane (like yours appears to be). Depending on how your control plane
>> is defined, the failure of one control node should be tolerable.
>> There has been some work on making compute nodes highly available but
>> I don't know the current status.
>
>
> could you point out the links/docs where I can refer for a proper setup.

you could check out the basic Openstack HA docs [2] which are outdated  
but can help to get a better understanding. It also contains a short  
section about HA compute nodes [3] which basically points to different  
sources, maybe check out [4], it also covers some basic aspects and  
refers to some possible solutions. But as I already wrote, that  
information is old, you'll need to follow up on Masakari or one of the  
other solutions presented there to see what the current status is.  
Because the HA docs [2] are not really detailed we had to collect most  
of the required information ourselves and I wrote some of it down in a  
blog post [5].

>> As for ceph it should be configured properly to sustain the loss of a
>> defined number of nodes or disks, I don't know your requirements. If
>> your current cluster has "only" 3 nodes you probably run replicated
>> pools with size 3 (I hope) with min_size 2 and failure-domain host.
>
> you mean 3 OSD s  in  single compute node ? I can follow this way is it the
> best way to do so ?

No, I don't talk about compute nodes here but ceph. You only mention  
that you have three ceph nodes, we don't more so it's difficult to  
comment.

>> If a compute node fails and is
>> unresponsive you'll probably need some DB tweaking to revive VMs on a
>> different node.
>>
> Don't know much about this, any reference is welcome.

I don't have a reference but only my blog post again [6] which  
contains an example, but I advise against tweaking the database if you  
don't know what you're doing just because a blog post says so. ;-)

There are so many aspects to consider, maybe you should consider  
professional consulting?

Regards,
Eugen


[2] https://docs.openstack.org/ha-guide/intro-ha.html
[3] https://docs.openstack.org/ha-guide/compute-node-ha.html
[4] http://aspiers.github.io/openstack-summit-2016-austin-compute-ha/#/agenda
[5]  
https://heiterbiswolkig.blogs.nde.ag/2020/08/21/openstack-ha-upgrade-part1/
[6]  
https://heiterbiswolkig.blogs.nde.ag/2020/09/25/openstack-ha-upgrade-part-v/


Zitat von KK CHN <kkchn.in at gmail.com>:

> Thanks Eugen .
> I fully agree with not running VMs on Control nodes.   When we rolled out
> the Controller resources we couldn't spare out only as a controller, Becoz
> of the utilization of the controller host machines resources, so we decided
> to use them as compute nodes also.
>
> On Thu, May 5, 2022 at 6:09 PM Eugen Block <eblock at nde.ag> wrote:
>
>> Hi,
>>
>> first, I wouldn't run VMs on control nodes, that way you mix roles
>> (control and compute) and in case that one control node fails the VMs
>> are not available. That would not be the case if the control node is
>> only a control node and is also part of a highly available control
>> plane (like yours appears to be). Depending on how your control plane
>> is defined, the failure of one control node should be tolerable.
>> There has been some work on making compute nodes highly available but
>> I don't know the current status.
>
>
> could you point out the links/docs where I can refer for a proper setup.
>
>
>> But in case a compute node fails but
>> nova is still responsive a live or cold migration could still be
>> possible to evacuate that host.
>
>
>
>> If a compute node fails and is
>> unresponsive you'll probably need some DB tweaking to revive VMs on a
>> different node.
>>
> Don't know much about this, any reference is welcome.
>
>
>> So you should have some spare resources to be able to recover from a
>> compute node failure.
>> As for ceph it should be configured properly to sustain the loss of a
>> defined number of nodes or disks, I don't know your requirements. If
>> your current cluster has "only" 3 nodes you probably run replicated
>> pools with size 3 (I hope) with min_size 2 and failure-domain host.
>
> you mean 3 OSD s  in  single compute node ? I can follow this way is it the
> best way to do so ?
>
>>
>>
> Any reference to this best ceph deployment model which do the best fault
> tolerance, kindly share.
>
>
>> You could sustain one node failure without clients noticing it, a
>> second node would cause the cluster to pause. Also you don't have the
>> possibility to recover from a node failure until it is up again,
>> meaning the degraded PGs can't be recovered on a different node. So
>> this also depends on your actual resiliency requirements. If you have
>> a second site you could use rbd mirroring [1] to sync all rbd images
>> between sites.
>
>
>  We have a connectivity link of only 1Gbps between DC and DR and  DR is 300
> miles away from DC.  And the syncing enhancement, how can we achieve this ?
> Because our HDD writing speeds are too limited, maybe 80 Mbps to 100 Mbps
> .. SSDs are not available for all compute host machines.
> Each VM has 800 GB to 1 TB Disk size.
>
> Is there any best practice for syncing enhancement mechanisms  for HDDs in
> DC hosts (with a  connectivity of 1 Gbps between DR sites with HDD hosts .
> ?)
>
>
>> In case the primary site goes down entirely you could
>> switch to the primary site by promoting the rbd images.
>> So you see there is plenty of information to cover and careful
>> planning is required.
>>
>> Regards,
>> Eugen
>>
>
> Thanks again for sharing your thoughts.
>
>>
>> [1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
>>
>> Zitat von KK CHN <kkchn.in at gmail.com>:
>>
>> > List,
>> >
>> > We are having an old cloud setup with OpenStack  Ussuri usng Debian OS,
>> > (Qemu KVM ).  I know its very old and we can't upgrade to to new versions
>> > right now.
>> >
>> > The  Deployment is as follows.
>> >
>> > A.    3 Controller in (cum compute nodes . VMs are running on controllers
>> > too..) in HA mode.
>> >
>> > B.   6 separate Compute nodes
>> >
>> > C.    3 separate Storage node with Ceph RBD
>> >
>> > Question is
>> >
>> > 1.  In case of any Sudden Hardware failure of one  or more controller
>> node
>> > OR Compute node  OR Storage Node  what will be the immediate redundant
>> > recovery setup need to be employed ?
>> >
>> > 2.  In case H/W failure our  recovery need to as soon as possible. For
>> > example less than30 Minutes after the first failure occurs.
>> >
>> > 3.  Is there setup options like a hot standby or similar setups or what
>> we
>> > need to employ ?
>> >
>> > 4. To meet all   RTO (< 30 Minutes down time ) and RPO(from the exact
>> point
>> > of crash all applications and data must be consistent) .
>> >
>> > 5. Please share  your thoughts for reliable crash/fault resistance
>> > configuration options in DC.
>> >
>> >
>> > We  have   a remote DR setup right now in a remote location. Also I would
>> > like to know if there is a recommended way to make the remote DR site
>> > Automatically up and run  ? OR How to automate the service from DR site
>> > to  meet exact RTO and RPO
>> >
>> > Any thoughts most welcom.
>> >
>> > Regards,
>> > Krish
>>
>>
>>
>>
>>