On 11/1/22 11:01, Swogat Pradhan wrote:
Hi, Updating the subject.
On Tue, Nov 1, 2022 at 12:26 PM Swogat Pradhan <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> wrote:
I have configured a 3 node pcs cluster for openstack. To test the HA, i issue the following commands: iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT && iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT && iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT && iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT && iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited && iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT && iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT && iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT && iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited
When i issue iptables command on 1 node then it is fenced and forced to reboot and cluster works fine. But when i issue this on 2 of the controller nodes the resource bundles fail and doesn't come back up.
This is expected behavior. In a cluster you need a majority quorum to be able to make the decision to fence a failing node, and keep services running on the nodes with the majority quorum. When you disconnect two nodes from the cluster with firewall rules, none of the 3 nodes can talk to any other node, i.e they are all isolated with no knowledge on what is the status on the 2 peer cluster nodes. Each node can only assume it is the only node that has been isolated, and the two other nodes are operational. To ensure data integrity any isolated node should stop it's services immediately. Imagine if all three nodes, isolated from each-other but still available to the loadbalancer. Requests would come in and each node would continue to service requests and write data. Each node servicing ~1/3 of the requests, the result would be a inconsistent data stores on all three nodes. A situation that would be practically impossible to recover from. -- Harald