<div dir="ltr"><div dir="ltr">Hi, </div><div>Updating the subject.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 1, 2022 at 12:26 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p id="m_-1077874144480820714gmail-yui_3_10_3_1_1667285511618_1472" style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">I have configured a 3 node pcs cluster for openstack.<br>To test the HA, i issue the following commands:<br>iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT &&<br>iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT &&<br>iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT &&<br>iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT &&<br>iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited &&<br>iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT &&<br>iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT &&<br>iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT &&<br>iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">When i issue iptables command on 1 node then it is fenced and forced to reboot and cluster works fine.<br>But when i issue this on 2 of the controller nodes the resource bundles fail and doesn't come back up.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">[root@overcloud-controller-1 ~]# pcs status<br>Cluster name: tripleo_cluster<br>Cluster Summary:<br> * Stack: corosync<br> * Current DC: overcloud-controller-1 (version 2.1.2-4.el8-ada5c3b36e2) - partition WITHOUT quorum<br> * Last updated: Sat Oct 29 03:15:29 2022<br> * Last change: Sat Oct 29 03:12:26 2022 by root via crm_resource on overcloud-controller-1<br> * 19 nodes configured<br> * 68 resource instances configured</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Node List:<br> * Node overcloud-controller-0: UNCLEAN (offline)<br> * Node overcloud-controller-2: UNCLEAN (offline)<br> * Online: [ overcloud-controller-1 ]</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Full List of Resources:<br> * ip-172.25.201.91 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN)<br> * ip-172.25.201.150 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN)<br> * ip-172.25.201.206 (ocf::heartbeat:IPaddr2): Stopped<br> * ip-172.25.201.250 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN)<br> * ip-172.25.202.50 (ocf::heartbeat:IPaddr2): Stopped<br> * ip-172.25.202.90 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN)<br> * Container bundle set: haproxy-bundle [<a href="http://172.25.201.68:8787/tripleomaster/openstack-haproxy:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-haproxy:pcmklatest</a>]:<br> * haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN)<br> * haproxy-bundle-podman-1 (ocf::heartbeat:podman): Stopped<br> * haproxy-bundle-podman-2 (ocf::heartbeat:podman): Started overcloud-controller-2 (UNCLEAN)<br> * haproxy-bundle-podman-3 (ocf::heartbeat:podman): Stopped<br> * Container bundle set: galera-bundle [<a href="http://172.25.201.68:8787/tripleomaster/openstack-mariadb:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-mariadb:pcmklatest</a>]:<br> * galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-controller-0 (UNCLEAN)<br> * galera-bundle-1 (ocf::heartbeat:galera): Stopped<br> * galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-2 (UNCLEAN)<br> * galera-bundle-3 (ocf::heartbeat:galera): Stopped<br> * Container bundle set: redis-bundle [<a href="http://172.25.201.68:8787/tripleomaster/openstack-redis:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-redis:pcmklatest</a>]:<br> * redis-bundle-0 (ocf::heartbeat:redis): Stopped<br> * redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-2 (UNCLEAN)<br> * redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-0 (UNCLEAN)<br> * redis-bundle-3 (ocf::heartbeat:redis): Stopped<br> * Container bundle set: ovn-dbs-bundle [<a href="http://172.25.201.68:8787/tripleomaster/openstack-ovn-northd:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-ovn-northd:pcmklatest</a>]:<br> * ovn-dbs-bundle-0 (ocf::ovn:ovndb-servers): Stopped overcloud-controller-2 (UNCLEAN)<br> * ovn-dbs-bundle-1 (ocf::ovn:ovndb-servers): Stopped overcloud-controller-0 (UNCLEAN)<br> * ovn-dbs-bundle-2 (ocf::ovn:ovndb-servers): Stopped<br> * ovn-dbs-bundle-3 (ocf::ovn:ovndb-servers): Stopped<br> * ip-172.25.201.208 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN)<br> * Container bundle: openstack-cinder-backup [<a href="http://172.25.201.68:8787/tripleomaster/openstack-cinder-backup:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-cinder-backup:pcmklatest</a>]:<br> * openstack-cinder-backup-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN)<br> * Container bundle: openstack-cinder-volume [<a href="http://172.25.201.68:8787/tripleomaster/openstack-cinder-volume:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-cinder-volume:pcmklatest</a>]:<br> * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Stopped<br> * Container bundle set: rabbitmq-bundle [<a href="http://172.25.201.68:8787/tripleomaster/openstack-rabbitmq:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-rabbitmq:pcmklatest</a>]:<br> * rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-2 (UNCLEAN)<br> * rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 (UNCLEAN)<br> * rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped<br> * rabbitmq-bundle-3 (ocf::heartbeat:rabbitmq-cluster): Stopped<br> * ip-172.25.204.250 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN)<br> * ceph-nfs (systemd:ceph-nfs@pacemaker): Started overcloud-controller-0 (UNCLEAN)<br> * Container bundle: openstack-manila-share [<a href="http://172.25.201.68:8787/tripleomaster/openstack-manila-share:pcmklatest" target="_blank">172.25.201.68:8787/tripleomaster/openstack-manila-share:pcmklatest</a>]:<br> * openstack-manila-share-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN)<br> * stonith-fence_ipmilan-48d539a11820 (stonith:fence_ipmilan): Stopped<br> * stonith-fence_ipmilan-48d539a1188c (stonith:fence_ipmilan): Started overcloud-controller-2 (UNCLEAN)<br> * stonith-fence_ipmilan-246e96349068 (stonith:fence_ipmilan): Started overcloud-controller-2 (UNCLEAN)<br> * stonith-fence_ipmilan-246e96348d30 (stonith:fence_ipmilan): Stopped</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Daemon Status:<br> corosync: active/enabled<br> pacemaker: active/enabled<br> pcsd: active/enabled</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">PCS requires more than half the nodes to be alive for the cluster to work. To fix this step I issued a command:<b>pcs no-quorum-policy=ignore.</b></p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">And now the PCS cluster keeps on running even when there is no quorum.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Now the issue i have is the mariadb-bundle becomes slave and dosen't get promoted to master.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Can you please suggest a proper workaround when more than half nodes go down and my cloud will be still running.</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px"><br></p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">With regards,</p><p style="margin:0px 0px 1.2em;padding:0px;width:auto;max-width:45em;color:rgb(51,51,51);font-family:monospace;font-size:12px">Swogat Pradhan</p></div>
</blockquote></div></div>