Hi Davorin,

have you checked a reference from RedHat Support page [1] . It looks like the situation is exactly the same as your issue.

/Jan Wasilewski

Reference:
[1] https://access.redhat.com/solutions/6991745 

wt., 7 sty 2025 o 12:52 <davorin.mikulic@gmail.com> napisał(a):
Dear Openstack community,

I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully.

Deploy fails on step 2 with:
2025-01-07 12:07:31.485297 |                                      |    WARNING | ERROR: Can't run container mysql_wait_bundle
stderr: + STEP=2
+ TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user
+ CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle'
+ EXTRA_ARGS=
+ '[' -d /tmp/puppet-etc ']'
+ cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ echo '{"step": 2}'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
+ set +e
+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle'
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
   (file & line not available)
Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies
Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies
Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies
Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2025-01-07 12:07:31.488412 |                                      |    WARNING | ERROR: Can't run container rabbitmq_wait_bundle
stderr: + STEP=2
+ TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready
+ CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle'
+ EXTRA_ARGS=
+ '[' -d /tmp/puppet-etc ']'
+ cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ echo '{"step": 2}'
+ export FACTER_deployment_type=containers
+ FACTER_deployment_type=containers
+ set +e
+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle'
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
   (file & line not available)
Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0]
Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies
+ rc=6
+ set -e
+ set +ux
2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"}
2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 |     TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
overcloud-cephstorage-0    : ok=273  changed=70   unreachable=0    failed=0    skipped=151  rescued=0    ignored=1
overcloud-cephstorage-1    : ok=275  changed=70   unreachable=0    failed=0    skipped=146  rescued=0    ignored=1
overcloud-cephstorage-2    : ok=275  changed=70   unreachable=0    failed=0    skipped=146  rescued=0    ignored=1
overcloud-computesriov-0   : ok=342  changed=92   unreachable=0    failed=0    skipped=163  rescued=0    ignored=1
overcloud-computesriov-1   : ok=349  changed=92   unreachable=0    failed=0    skipped=156  rescued=0    ignored=1
overcloud-controller-0     : ok=372  changed=99   unreachable=0    failed=1    skipped=217  rescued=0    ignored=1
overcloud-controller-1     : ok=392  changed=98   unreachable=0    failed=0    skipped=212  rescued=0    ignored=1
overcloud-controller-2     : ok=392  changed=98   unreachable=0    failed=0    skipped=212  rescued=0    ignored=1
undercloud                 : ok=76   changed=19   unreachable=0    failed=0    skipped=31   rescued=0    ignored=1
2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.563565 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 |    SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s
2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c |    SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s
2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e |    SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s
2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s
2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s
2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s
2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s
2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 |    SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s
2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s
2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s
2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 |    SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s
2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 |    SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s
2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 |    SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s
2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 |    SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s
2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad |    SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s
2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad |    SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s
2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 |    SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s
2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 |    SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s
2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 |    SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s
2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 |    SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s
2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2025-01-07 12:07:31.577328 |  The following node(s) had failures: overcloud-controller-0
2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up
2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts
Overcloud Endpoint: http://10.100.27.127:5000
Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard
Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc
Overcloud Deployed with error
2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.set_deployment_status(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.config_download(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     utils.run_ansible_playbook(
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise RuntimeError(err_msg)
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1


I can see the following logs on the failed controller-0 node:
[root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log
** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56 =ERROR REPORT====
** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:29:03 =ERROR REPORT====
** Generic server rabbit_disk_monitor terminating
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000}
** Reason for termination ==
** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2025-01-07 10:29:03 =CRASH REPORT====
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.309.0>
    registered_name: rabbit_disk_monitor
    exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>]
    message_queue_len: 0
    messages: []
    links: [<0.308.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 314108420
  neighbours:
2025-01-07 10:29:03 =SUPERVISOR REPORT====
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    child_terminated
     Reason:     {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}
     Offender:   [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}]

less /var/log/containers/rabbitmq/rabbit@overcloud-controller-0.internalapi.tb27.cld.log
2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' **
2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f
alse,true,10,120000}
** Reason for termination ==
** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr
c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r
abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini
t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou
d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272
2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit
with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated
2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10
2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9
2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq
/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8

[root@overcloud-controller-0 ~]# pcs status --full
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum
  * Last updated: Tue Jan  7 11:49:48 2025 on overcloud-controller-0
  * Last change:  Tue Jan  7 10:33:11 2025 by root via root on overcloud-controller-0
  * 9 nodes configured
  * 26 resource instances configured (1 BLOCKED from further action due to failure)

Node List:
  * GuestNode galera-bundle-0@overcloud-controller-2: online
  * GuestNode galera-bundle-1@overcloud-controller-0: online
  * GuestNode galera-bundle-2@overcloud-controller-1: online
  * Node overcloud-controller-0 (1): online, feature set 3.19.6
  * Node overcloud-controller-1 (2): online, feature set 3.19.6
  * Node overcloud-controller-2 (3): online, feature set 3.19.6
  * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online
  * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online
  * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online

Full List of Resources:
  * ip-10.100.21.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-0
  * ip-10.100.27.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-1
  * ip-10.100.22.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-2
  * ip-10.100.24.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-0
  * ip-10.100.25.127    (ocf:heartbeat:IPaddr2):         Started overcloud-controller-1
  * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]:
    * Replica[0]
      * haproxy-bundle-podman-0 (ocf:heartbeat:podman):  Started overcloud-controller-0
    * Replica[1]
      * haproxy-bundle-podman-1 (ocf:heartbeat:podman):  Started overcloud-controller-1
    * Replica[2]
      * haproxy-bundle-podman-2 (ocf:heartbeat:podman):  Started overcloud-controller-2
  * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]:
    * Replica[0]
      * galera  (ocf:heartbeat:galera):  FAILED Promoted galera-bundle-0 (blocked)
      * galera-bundle-podman-0  (ocf:heartbeat:podman):  Started overcloud-controller-2
      * galera-bundle-0 (ocf:pacemaker:remote):  Started overcloud-controller-2
    * Replica[1]
      * galera  (ocf:heartbeat:galera):  Unpromoted galera-bundle-1
      * galera-bundle-podman-1  (ocf:heartbeat:podman):  Started overcloud-controller-0
      * galera-bundle-1 (ocf:pacemaker:remote):  Started overcloud-controller-0
    * Replica[2]
      * galera  (ocf:heartbeat:galera):  Unpromoted galera-bundle-2
      * galera-bundle-podman-2  (ocf:heartbeat:podman):  Started overcloud-controller-1
      * galera-bundle-2 (ocf:pacemaker:remote):  Started overcloud-controller-1
  * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]:
    * Replica[0]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-0
      * rabbitmq-bundle-podman-0        (ocf:heartbeat:podman):  Started overcloud-controller-2
      * rabbitmq-bundle-0       (ocf:pacemaker:remote):  Started overcloud-controller-2
    * Replica[1]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-1
      * rabbitmq-bundle-podman-1        (ocf:heartbeat:podman):  Started overcloud-controller-0
      * rabbitmq-bundle-1       (ocf:pacemaker:remote):  Started overcloud-controller-0
    * Replica[2]
      * rabbitmq        (ocf:heartbeat:rabbitmq-cluster):        Started rabbitmq-bundle-2
      * rabbitmq-bundle-podman-2        (ocf:heartbeat:podman):  Started overcloud-controller-1
      * rabbitmq-bundle-2       (ocf:pacemaker:remote):  Started overcloud-controller-1

Node Attributes:
  * Node: overcloud-controller-0 (1):
    * galera-last-committed             : -1
    * galera-no-grastate                : true
    * galera-role                       : true
    * haproxy-role                      : true
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-0.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-0.internalapi.tb27.cld
  * Node: overcloud-controller-1 (2):
    * galera-last-committed             : -1
    * galera-no-grastate                : true
    * galera-role                       : true
    * haproxy-role                      : true
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-1.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-1.internalapi.tb27.cld
  * Node: overcloud-controller-2 (3):
    * galera-bootstrap                  : true
    * galera-role                       : true
    * haproxy-role                      : true
    * master-galera                     : 100
    * rabbitmq-role                     : true
    * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-2.internalapi.tb27.cld
    * rmq-node-attr-rabbitmq            : rabbit@overcloud-controller-2.internalapi.tb27.cld

Migration Summary:
  * Node: galera-bundle-0@overcloud-controller-2:
    * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan  7 10:32:38 2025'

Failed Resource Actions:
  * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan  7 10:32:28 2025', queued=0ms, exec=10603ms

Tickets:

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


Does someone perhaps knows where the issue lies?

Thank you very much!