Hi Davorin,

Have you removed the overcloud stack before? Please double-check that your MySQL container is also removed. It seems like it is still using the password from the first deployment (or it wasn't set correctly). Especially since you confirmed that everything works as expected for another deployment.

/Jan Wasilewski

śr., 8 sty 2025 o 10:06 Davorin Mikulić <davorin.mikulic@gmail.com> napisał(a):
Hello,

I indeed checked the link you provided earlier, but the  overcloud_domain_name and CloudDomain are completely the same. I also found this link: https://access.redhat.com/solutions/7060294 but this did not resolve the problem as well.

I tried manually entering the mysql DB with:
[root@overcloud-controller-2 ~]# cat /var/lib/config-data/puppet-generated/mysql/root/.my.cnf
[client]
user=root
password="L7W48B553P"

[mysql]
user=root
password="L7W48B553P"[root@overcloud-controller-2 ~]#
[root@overcloud-controller-2 ~]#
[root@overcloud-controller-2 ~]#
[root@overcloud-controller-2 ~]#
[root@overcloud-controller-2 ~]# podman exec -it 26c44c007caa bash
bash-4.4$ netstat -anp | grep 3306
tcp        0      0 10.100.22.89:3306       0.0.0.0:*               LISTEN      -
tcp        0      0 10.100.22.127:3306      0.0.0.0:*               LISTEN      -
bash-4.4$ mysql -u root -pL7W48B553P
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
bash-4.4$ mysql -u root -pL7W48B553P -h 10.100.22.89
ERROR 1045 (28000): Access denied for user 'root'@'10.100.22.89' (using password: YES)
bash-4.4$ mysql -u root -pL7W48B553P -h 10.100.22.127
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11
bash-4.4$ mysql -u root -p'L7W48B553P' -h 10.100.22.127
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11
bash-4.4$ exit
[root@overcloud-controller-2 ~]# hiera -c /etc/puppet/hiera.yaml mysql::server::root_password
L7W48B553P
[root@overcloud-controller-2 ~]# cat /etc/sysconfig/clustercheck
cat: /etc/sysconfig/clustercheck: No such file or directory
[root@overcloud-controller-2 ~]# podman exec -it 26c44c007caa bash
bash-4.4$ cat /etc/sysconfig/clustercheck
MYSQL_USERNAME=clustercheck

MYSQL_PASSWORD='ERgHdKeisDLwmj9r5EG0ApeZr'

MYSQL_HOST=localhost

TRIPLEO_SOCAT_BIND='tcp4-listen:9200,bind="10.100.22.89",reuseaddr,fork'

TRIPLEO_POST_STATUS_WAIT=0
bash-4.4$ mysql -u clustercheck -pERgHdKeisDLwmj9r5EG0ApeZr -h 10.100.22.127
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11
bash-4.4$ mysql -u clustercheck -pERgHdKeisDLwmj9r5EG0ApeZr -h localhost
ERROR 1045 (28000): Access denied for user 'clustercheck'@'localhost' (using password: YES)
bash-4.4$ mysql -u clustercheck -p'ERgHdKeisDLwmj9r5EG0ApeZr' -h localhost
ERROR 1045 (28000): Access denied for user 'clustercheck'@'localhost' (using password: YES)
bash-4.4$ mysql -u clustercheck -p'ERgHdKeisDLwmj9r5EG0ApeZr' -h localhost:9200
ERROR 2005 (HY000): Unknown MySQL server host 'localhost:9200' (-2)
bash-4.4$

But none of it seems to work.

On Wed, Jan 8, 2025 at 9:56 AM Jan Wasilewski <finarffin@gmail.com> wrote:
Hi Davorin,

have you checked a reference from RedHat Support page [1] . It looks like the situation is exactly the same as your issue.

/Jan Wasilewski

Reference:

wt., 7 sty 2025 o 12:52 <davorin.mikulic@gmail.com> napisał(a):
Dear Openstack community,

I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully.

Deploy fails on step 2 with:
2025-01-07 12:07:31.485297 |                                      |    WARNING | ERROR: Can't run container mysql_wait_bundle
stderr: + STEP=2
+ TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user
+ CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle'
+ EXTRA_ARGS=
+ '[' -d /tmp/puppet-etc ']'
+ cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ echo '{"step": 2}'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
+ set +e
+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle'
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
   (file & line not available)
Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies
Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies
Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2025-01-07 12:07:31.488412 |                                      |    WARNING | ERROR: Can't run container rabbitmq_wait_bundle
stderr: + STEP=2
+ TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready
+ CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle'
+ EXTRA_ARGS=
+ '[' -d /tmp/puppet-etc ']'
+ cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ echo '{"step": 2}'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
+ set +e
+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle'
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
   (file & line not available)
Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0]
Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"}
2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 |     TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
overcloud-cephstorage-0    : ok=273  changed=70   unreachable=0    failed=0    skipped=151  rescued=0    ignored=1
overcloud-cephstorage-1    : ok=275  changed=70   unreachable=0    failed=0    skipped=146  rescued=0    ignored=1
overcloud-cephstorage-2    : ok=275  changed=70   unreachable=0    failed=0    skipped=146  rescued=0    ignored=1
overcloud-computesriov-0   : ok=342  changed=92   unreachable=0    failed=0    skipped=163  rescued=0    ignored=1
overcloud-computesriov-1   : ok=349  changed=92   unreachable=0    failed=0    skipped=156  rescued=0    ignored=1
overcloud-controller-0     : ok=372  changed=99   unreachable=0    failed=1    skipped=217  rescued=0    ignored=1
overcloud-controller-1     : ok=392  changed=98   unreachable=0    failed=0    skipped=212  rescued=0    ignored=1
overcloud-controller-2     : ok=392  changed=98   unreachable=0    failed=0    skipped=212  rescued=0    ignored=1
undercloud                 : ok=76   changed=19   unreachable=0    failed=0    skipped=31   rescued=0    ignored=1
2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.563565 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 |    SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s
2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c |    SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s
2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e |    SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s
2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s
2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s
2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s
2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s
2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s
2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s
2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s
2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s
2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 |    SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s
2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 |    SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s
2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 |    SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s
2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad |    SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s
2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad |    SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s
2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 |    SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s
2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 |    SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s
2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 |    SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s
2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 |    SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s
2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.577328 |  The following node(s) had failures: overcloud-controller-0
2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up
2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts
Overcloud Endpoint: http://10.100.27.127:5000
Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard
Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc
Overcloud Deployed with error
2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.set_deployment_status(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.config_download(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     utils.run_ansible_playbook(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise RuntimeError(err_msg)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1


I can see the following logs on the failed controller-0 node:
[root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log
** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:29:03 =ERROR REPORT====
** Generic server rabbit_disk_monitor terminating
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000}
** Reason for termination ==
** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2025-01-07 10:29:03 =CRASH REPORT====
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.309.0>
    registered_name: rabbit_disk_monitor
    exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>]
    message_queue_len: 0
    messages: []
    links: [<0.308.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 314108420
  neighbours:
2025-01-07 10:29:03 =SUPERVISOR REPORT====
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    child_terminated
     Reason:     {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}
     Offender:   [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}]

less /var/log/containers/rabbitmq/rabbit@overcloud-controller-0.internalapi.tb27.cld.log
2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f
alse,true,10,120000}
** Reason for termination ==
** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr
c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r
abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini
t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou
d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272
2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit
with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated
2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10
2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9
2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8

[root@overcloud-controller-0 ~]# pcs status --full
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum
  * Last updated: Tue Jan  7 11:49:48 2025 on overcloud-controller-0
  * Last change:  Tue Jan  7 10:33:11 2025 by root via root on overcloud-controller-0
  * 9 nodes configured
  * 26 resource instances configured (1 BLOCKED from further action due to failure)

Node List:
  * GuestNode galera-bundle-0@overcloud-controller-2: online
  * GuestNode galera-bundle-1@overcloud-controller-0: online
  * GuestNode galera-bundle-2@overcloud-controller-1: online
  * Node overcloud-controller-0 (1): online, feature set 3.19.6
  * Node overcloud-controller-1 (2): online, feature set 3.19.6
  * Node overcloud-controller-2 (3): online, feature set 3.19.6
  * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online
  * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online
  * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online

Full List of Resources:
  * ip-10.100.21.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-0
  * ip-10.100.27.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-1
  * ip-10.100.22.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-2
  * ip-10.100.24.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-0
  * ip-10.100.25.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-1
  * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]:
    * Replica[0]
      * haproxy-bundle-podman-0 (ocf:heartbeat:podman):  Started overcloud-controller-0
    * Replica[1]
      * haproxy-bundle-podman-1 (ocf:heartbeat:podman):  Started overcloud-controller-1
    * Replica[2]
      * haproxy-bundle-podman-2 (ocf:heartbeat:podman):  Started overcloud-controller-2
  * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]:
    * Replica[0]
      * galera  (ocf:heartbeat:galera):  FAILED Promoted galera-bundle-0 (blocked)
      * galera-bundle-podman-0  (ocf:heartbeat:podman):  Started overcloud-controller-2
      * galera-bundle-0 (ocf:pacemaker:remote):  Started overcloud-controller-2
    * Replica[1]
      * galera  (ocf:heartbeat:galera):  Unpromoted galera-bundle-1
      * galera-bundle-podman-1  (ocf:heartbeat:podman):  Started overcloud-controller-0
      * galera-bundle-1 (ocf:pacemaker:remote):  Started overcloud-controller-0
    * Replica[2]
      * galera  (ocf:heartbeat:galera):  Unpromoted galera-bundle-2
      * galera-bundle-podman-2  (ocf:heartbeat:podman):  Started overcloud-controller-1
      * galera-bundle-2 (ocf:pacemaker:remote):  Started overcloud-controller-1
  * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]:
    * Replica[0]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-0
      * rabbitmq-bundle-podman-0        (ocf:heartbeat:podman):  Started overcloud-controller-2
      * rabbitmq-bundle-0       (ocf:pacemaker:remote):  Started overcloud-controller-2
    * Replica[1]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-1
      * rabbitmq-bundle-podman-1        (ocf:heartbeat:podman):  Started overcloud-controller-0
      * rabbitmq-bundle-1       (ocf:pacemaker:remote):  Started overcloud-controller-0
    * Replica[2]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-2
      * rabbitmq-bundle-podman-2        (ocf:heartbeat:podman):  Started overcloud-controller-1
      * rabbitmq-bundle-2       (ocf:pacemaker:remote):  Started overcloud-controller-1

Node Attributes:
  * Node: overcloud-controller-0 (1):
    * galera-last-committed             : -1
    * galera-no-grastate                : true
    * galera-role                       : true
    * haproxy-role                      : true
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-0.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-0.internalapi.tb27.cld
  * Node: overcloud-controller-1 (2):
    * galera-last-committed             : -1
    * galera-no-grastate                : true
    * galera-role                       : true
    * haproxy-role                      : true
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-1.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-1.internalapi.tb27.cld
  * Node: overcloud-controller-2 (3):
    * galera-bootstrap                  : true
    * galera-role                       : true
    * haproxy-role                      : true
    * master-galera                     : 100
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-2.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-2.internalapi.tb27.cld

Migration Summary:
  * Node: galera-bundle-0@overcloud-controller-2:
    * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan  7 10:32:38 2025'

Failed Resource Actions:
  * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan  7 10:32:28 2025', queued=0ms, exec=10603ms

Tickets:

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


Does someone perhaps knows where the issue lies?

Thank you very much!