Dear Openstack community, I am trying to deploy Wallaby overcloud on HP Gen9 machines with CentOS 9 installed on them. The very similar setup with CentOS 8 deployed and worked successfully. Deploy fails on step 2 with: 2025-01-07 12:07:31.485297 | | WARNING | ERROR: Can't run container mysql_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user + CONFIG='include tripleo::profile::pacemaker::database::mysql_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::database::mysql_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@::1]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@%]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@localhost.localdomain]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0.tb27.cld]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@overcloud-controller-0]: Skipping because of failed dependencies Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_user[@overcloud-controller-0]: Skipping because of failed dependencies Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Mysql::Server::Account_security/Mysql_database[test]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[root@%]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Mysql_user[clustercheck@localhost]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Mysql_database[nova_cell0]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Mysql_database[nova_api]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Mysql_database[placement]: Skipping because of failed dependencies Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova_cell0]/Openstacklib::Db::Mysql::Host_access[nova_cell0_%]/Mysql_grant[nova@%/nova_cell0.*]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_user[nova_api@%]: Skipping because of failed dependencies Warning: /Stage[main]/Nova::Db::Mysql_api/Openstacklib::Db::Mysql[nova_api]/Openstacklib::Db::Mysql::Host_access[nova_api_%]/Mysql_grant[nova_api@%/nova_api.*]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_user[placement@%]: Skipping because of failed dependencies Warning: /Stage[main]/Placement::Db::Mysql/Openstacklib::Db::Mysql[placement]/Openstacklib::Db::Mysql::Host_access[placement_%]/Mysql_grant[placement@%/placement.*]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.488412 | | WARNING | ERROR: Can't run container rabbitmq_wait_bundle stderr: + STEP=2 + TAGS=file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready + CONFIG='include tripleo::profile::pacemaker::rabbitmq_bundle' + EXTRA_ARGS= + '[' -d /tmp/puppet-etc ']' + cp -a /tmp/puppet-etc/devices /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet + echo '{"step": 2}' + export FACTER_deployment_type=containers + FACTER_deployment_type=containers + set +e + puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e 'noop_resource('\''package'\''); include tripleo::profile::pacemaker::rabbitmq_bundle' Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (file: /etc/puppet/hiera.yaml) Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html (file & line not available) Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6) Error: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/returns: change from 'notrun' to ['0'] failed: 'rabbitmqctl eval "rabbit_nodes:is_running(node(), rabbit)." | grep -q true && rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true' returned 1 instead of one of [0] Warning: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Skipping because of failed dependencies Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Rabbitmq_policy[ha-all@/]: Skipping because of failed dependencies + rc=6 + set -e + set +ux 2025-01-07 12:07:31.489548 | 000c29c7-8523-9a37-c0c4-00000000f883 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: mysql_wait_bundle, rabbitmq_wait_bundle"} 2025-01-07 12:07:31.490512 | 000c29c7-8523-9a37-c0c4-00000000f883 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | overcloud-controller-0 | 0:48:23.163306 | 1989.39s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=273 changed=70 unreachable=0 failed=0 skipped=151 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=275 changed=70 unreachable=0 failed=0 skipped=146 rescued=0 ignored=1 overcloud-computesriov-0 : ok=342 changed=92 unreachable=0 failed=0 skipped=163 rescued=0 ignored=1 overcloud-computesriov-1 : ok=349 changed=92 unreachable=0 failed=0 skipped=156 rescued=0 ignored=1 overcloud-controller-0 : ok=372 changed=99 unreachable=0 failed=1 skipped=217 rescued=0 ignored=1 overcloud-controller-1 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 overcloud-controller-2 : ok=392 changed=98 unreachable=0 failed=0 skipped=212 rescued=0 ignored=1 undercloud : ok=76 changed=19 unreachable=0 failed=0 skipped=31 rescued=0 ignored=1 2025-01-07 12:07:31.561753 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.562447 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1905 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563026 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:48:23.235822 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.563565 | UUID | Info | Host | Task Name | Run Time 2025-01-07 12:07:31.564127 | 000c29c7-8523-9a37-c0c4-00000000f883 | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_2 | 1989.39s 2025-01-07 12:07:31.564700 | 000c29c7-8523-9a37-c0c4-00000000bb0c | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 137.67s 2025-01-07 12:07:31.565400 | 000c29c7-8523-9a37-c0c4-00000000e34e | SUMMARY | undercloud | tripleo_run_cephadm : run cephadm playbook | 59.69s 2025-01-07 12:07:31.565937 | 000c29c7-8523-9a37-c0c4-00000000e715 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 56.48s 2025-01-07 12:07:31.566525 | 000c29c7-8523-9a37-c0c4-00000000e747 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 46.63s 2025-01-07 12:07:31.567080 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 36.87s 2025-01-07 12:07:31.567628 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 36.75s 2025-01-07 12:07:31.568206 | 000c29c7-8523-9a37-c0c4-000000004353 | SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 36.65s 2025-01-07 12:07:31.568767 | 000c29c7-8523-9a37-c0c4-00000000e776 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run pacemaker restart if the config file for the service changed | 36.45s 2025-01-07 12:07:31.569335 | 000c29c7-8523-9a37-c0c4-00000000e745 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 24.87s 2025-01-07 12:07:31.569886 | 000c29c7-8523-9a37-c0c4-00000000e774 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 23.94s 2025-01-07 12:07:31.570454 | 000c29c7-8523-9a37-c0c4-00000000caa9 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.83s 2025-01-07 12:07:31.571025 | 000c29c7-8523-9a37-c0c4-00000000f709 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 20.82s 2025-01-07 12:07:31.571574 | 000c29c7-8523-9a37-c0c4-00000000be26 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 15.90s 2025-01-07 12:07:31.572173 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 15.88s 2025-01-07 12:07:31.572740 | 000c29c7-8523-9a37-c0c4-00000000bdad | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 15.68s 2025-01-07 12:07:31.573331 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-1 | tripleo_hieradata : Render hieradata from template | 11.32s 2025-01-07 12:07:31.573885 | 000c29c7-8523-9a37-c0c4-0000000024d7 | SUMMARY | overcloud-computesriov-0 | tripleo_hieradata : Render hieradata from template | 11.14s 2025-01-07 12:07:31.574467 | 000c29c7-8523-9a37-c0c4-00000000c4a0 | SUMMARY | overcloud-computesriov-1 | Wait for puppet host configuration to finish | 11.07s 2025-01-07 12:07:31.575030 | 000c29c7-8523-9a37-c0c4-00000000c452 | SUMMARY | overcloud-computesriov-0 | Wait for puppet host configuration to finish | 10.98s 2025-01-07 12:07:31.575572 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576197 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.576756 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:31.577328 | The following node(s) had failures: overcloud-controller-0 2025-01-07 12:07:31.577879 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2025-01-07 12:07:32.979 180292 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleogr2nbip5 ] cleaned up 2025-01-07 12:07:32.979 180292 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:32.981 180292 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2025-01-07 12:07:33.690 180292 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2025-01-07 12:07:33.774 180292 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook( 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.431 180292 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2025-01-07 12:07:36.434 180292 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2025-01-07 12:07:36.437 180292 INFO osc_lib.shell [-] END return value: 1 I can see the following logs on the failed controller-0 node: [root@overcloud-controller-0 log]# less /var/log/containers/rabbitmq/log/crash.log ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56 =ERROR REPORT==== ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03 =ERROR REPORT==== ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,false,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03 =CRASH REPORT==== crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.309.0> registered_name: rabbit_disk_monitor exception exit: {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.301.0>] message_queue_len: 0 messages: [] links: [<0.308.0>] dictionary: [] trap_exit: false status: running heap_size: 2586 stack_size: 27 reductions: 314108420 neighbours: 2025-01-07 10:29:03 =SUPERVISOR REPORT==== Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} Offender: [{pid,<0.309.0>},{name,rabbit_disk_monitor},{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},{restart_type,{transient,1}},{shutdown,30000},{child_type,worker}] less /var/log/containers/rabbitmq/rabbit@overcloud-controller-0.internalapi.tb27.cld.log 2024-12-19 19:18:44.490 [info] <0.9.0> Server startup complete; 3 plugins started. * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_management_agent 2025-01-07 10:28:44.393 [error] <0.11277.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:44.404 [error] <0.11280.105> ** Connection attempt from disallowed node 'rabbitmqcli-169019-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.167 [error] <0.11284.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:45.179 [error] <0.11286.105> ** Connection attempt from disallowed node 'rabbitmqcli-169251-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.282 [error] <0.11282.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.293 [error] <0.11293.105> ** Connection attempt from disallowed node 'rabbitmqcli-169491-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.779 [error] <0.11298.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:46.788 [error] <0.11300.105> ** Connection attempt from disallowed node 'rabbitmqcli-169710-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.948 [error] <0.11303.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:52.958 [error] <0.11312.105> ** Connection attempt from disallowed node 'rabbitmqcli-169935-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.824 [error] <0.11317.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:54.837 [error] <0.11318.105> ** Connection attempt from disallowed node 'rabbitmqcli-170662-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.645 [error] <0.11325.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:55.655 [error] <0.11327.105> ** Connection attempt from disallowed node 'rabbitmqcli-170889-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.160 [error] <0.11328.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:28:56.171 [error] <0.11334.105> ** Connection attempt from disallowed node 'rabbitmqcli-171103-rabbit@overcloud-controller-0.internalapi.tb27.cld' ** 2025-01-07 10:29:03.051 [error] <0.309.0> ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld",50000000,1173484339200,100,10000,#Ref<0.112052652.1425801217.106754>,f alse,true,10,120000} ** Reason for termination == ** {{unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"},[{rabbit_disk_monitor,parse_free_unix,1,[{file,"sr c/rabbit_disk_monitor.erl"},{line,272}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,218}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/r abbit_disk_monitor.erl"},{line,190}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,ini t_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} 2025-01-07 10:29:03.051 [error] <0.309.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours exited with reason: {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overclou d-controller-0.internalapi.tb27.cld: No such file or directory\n"} in rabbit_disk_monitor:parse_free_unix/1 line 272 2025-01-07 10:29:03.051 [error] <0.308.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.309.0> exit with reason {unparseable,"df: /var/lib/rabbitmq/mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"} in context child_terminated 2025-01-07 10:29:03.059 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 10 2025-01-07 10:31:03.068 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 9 2025-01-07 10:33:03.076 [info] <0.11345.105> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{unparseable,"df: /var/lib/rabbitmq /mnesia/rabbit@overcloud-controller-0.internalapi.tb27.cld: No such file or directory\n"}},202138791936}, retries left: 8 [root@overcloud-controller-0 ~]# pcs status --full Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: overcloud-controller-1 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum * Last updated: Tue Jan 7 11:49:48 2025 on overcloud-controller-0 * Last change: Tue Jan 7 10:33:11 2025 by root via root on overcloud-controller-0 * 9 nodes configured * 26 resource instances configured (1 BLOCKED from further action due to failure) Node List: * GuestNode galera-bundle-0@overcloud-controller-2: online * GuestNode galera-bundle-1@overcloud-controller-0: online * GuestNode galera-bundle-2@overcloud-controller-1: online * Node overcloud-controller-0 (1): online, feature set 3.19.6 * Node overcloud-controller-1 (2): online, feature set 3.19.6 * Node overcloud-controller-2 (3): online, feature set 3.19.6 * GuestNode rabbitmq-bundle-0@overcloud-controller-2: online * GuestNode rabbitmq-bundle-1@overcloud-controller-0: online * GuestNode rabbitmq-bundle-2@overcloud-controller-1: online Full List of Resources: * ip-10.100.21.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.27.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * ip-10.100.22.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-2 * ip-10.100.24.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-0 * ip-10.100.25.127 (ocf:heartbeat:IPaddr2): Started overcloud-controller-1 * Container bundle set: haproxy-bundle [cluster.common.tag/haproxy:pcmklatest]: * Replica[0] * haproxy-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-0 * Replica[1] * haproxy-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-1 * Replica[2] * haproxy-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-2 * Container bundle set: galera-bundle [cluster.common.tag/mariadb:pcmklatest]: * Replica[0] * galera (ocf:heartbeat:galera): FAILED Promoted galera-bundle-0 (blocked) * galera-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * galera-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-1 * galera-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * galera-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * galera (ocf:heartbeat:galera): Unpromoted galera-bundle-2 * galera-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * galera-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 * Container bundle set: rabbitmq-bundle [cluster.common.tag/rabbitmq:pcmklatest]: * Replica[0] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-0 * rabbitmq-bundle-podman-0 (ocf:heartbeat:podman): Started overcloud-controller-2 * rabbitmq-bundle-0 (ocf:pacemaker:remote): Started overcloud-controller-2 * Replica[1] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-1 * rabbitmq-bundle-podman-1 (ocf:heartbeat:podman): Started overcloud-controller-0 * rabbitmq-bundle-1 (ocf:pacemaker:remote): Started overcloud-controller-0 * Replica[2] * rabbitmq (ocf:heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-2 * rabbitmq-bundle-podman-2 (ocf:heartbeat:podman): Started overcloud-controller-1 * rabbitmq-bundle-2 (ocf:pacemaker:remote): Started overcloud-controller-1 Node Attributes: * Node: overcloud-controller-0 (1): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-0.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit@overcloud-controller-0.internalapi.tb27.cld * Node: overcloud-controller-1 (2): * galera-last-committed : -1 * galera-no-grastate : true * galera-role : true * haproxy-role : true * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-1.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit@overcloud-controller-1.internalapi.tb27.cld * Node: overcloud-controller-2 (3): * galera-bootstrap : true * galera-role : true * haproxy-role : true * master-galera : 100 * rabbitmq-role : true * rmq-node-attr-last-known-rabbitmq : rabbit@overcloud-controller-2.internalapi.tb27.cld * rmq-node-attr-rabbitmq : rabbit@overcloud-controller-2.internalapi.tb27.cld Migration Summary: * Node: galera-bundle-0@overcloud-controller-2: * galera: migration-threshold=1000000 fail-count=1 last-failure='Tue Jan 7 10:32:38 2025' Failed Resource Actions: * galera_promote_0 on galera-bundle-0 'error' (1): call=116, status='complete', exitreason='Failed initial monitor action', last-rc-change='Tue Jan 7 10:32:28 2025', queued=0ms, exec=10603ms Tickets: PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Does someone perhaps knows where the issue lies? Thank you very much!