Hi, I have successfully deployed networks, vips, overcloud nodes and ceph cluster. Now I am proceeding with openstack overcloud deploy command as follows: openstack overcloud deploy --deployed-server --answers-file /home/stack/3XOenvironment/answers.yaml answers.yaml is as follows: (undercloud) [stack@uc4tb27 3XOenvironment]$ cat answers.yaml templates: /home/stack/generated-openstack-tripleo-heat-templates networks: /home/stack/3XOenvironment/40-network_data.yaml roles: /home/stack/3XOenvironment/30-roles_data.yaml environments: - /home/stack/3XOenvironment/deployed-server-environment.yaml - /home/stack/3XOenvironment/network-environment.yaml - /home/stack/3XOenvironment/overcloud-networks-deployed.yaml - /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml - /home/stack/3XOenvironment/deployed-vips.yaml - /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml - /home/stack/3XOenvironment/10-docker-ha.yaml - /home/stack/3XOenvironment/cephadm.yaml - /home/stack/3XOenvironment/deployed-ceph.yaml We are not using a TLS setup anywhere in the overcloud or the undercloud. During the deploy, after the stack has been successfully created and deploy proceeds to puppet configuration of containers in step 1, I am getting the following error: 2024-01-17 09:30:39.360378 | 000c29b1-9f06-28d0-4abe-00000000d389 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-1 | 0:18:04.150199 | 324.03s 2024-01-17 09:30:43.693815 | | WARNING | ERROR: Can't run container container-puppet-glance_api_internal stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:20 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:20 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:20 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:20 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-2.tb27.cld + rc=1 + '[' false = false ']' + set +x 2024-01-17 09:30:43.695467 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | error={"changed": false, "msg": "Failed containers: container-puppet-glance_api_internal"} 2024-01-17 09:30:43.696471 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | 0:18:08.486292 | 323.73s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=238 changed=96 unreachable=0 failed=0 skipped=135 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-controller-0 : ok=295 changed=140 unreachable=0 failed=1 skipped=175 rescued=0 ignored=1 overcloud-controller-1 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-controller-2 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-novacompute-0 : ok=281 changed=132 unreachable=0 failed=0 skipped=165 rescued=0 ignored=1 overcloud-novacompute-1 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 overcloud-novacompute-2 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 undercloud : ok=25 changed=9 unreachable=0 failed=0 skipped=2 rescued=0 ignored=1 2024-01-17 09:30:43.764701 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.765549 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1578 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766211 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:18:08.556038 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766837 | UUID | Info | Host | Task Name | Run Time 2024-01-17 09:30:43.767508 | 000c29b1-9f06-28d0-4abe-00000000d389 | SUMMARY | overcloud-controller-1 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 324.03s 2024-01-17 09:30:43.768148 | 000c29b1-9f06-28d0-4abe-00000000d33c | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.768828 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | SUMMARY | overcloud-controller-2 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.769558 | 000c29b1-9f06-28d0-4abe-00000000b974 | SUMMARY | overcloud-novacompute-0 | Pre-fetch all the containers | 181.76s 2024-01-17 09:30:43.770215 | 000c29b1-9f06-28d0-4abe-00000000b9f7 | SUMMARY | overcloud-novacompute-1 | Pre-fetch all the containers | 175.41s 2024-01-17 09:30:43.770929 | 000c29b1-9f06-28d0-4abe-00000000ba70 | SUMMARY | overcloud-novacompute-2 | Pre-fetch all the containers | 169.84s 2024-01-17 09:30:43.771586 | 000c29b1-9f06-28d0-4abe-00000000c777 | SUMMARY | overcloud-controller-2 | Wait for puppet host configuration to finish | 133.41s 2024-01-17 09:30:43.772286 | 000c29b1-9f06-28d0-4abe-00000000c730 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 133.31s 2024-01-17 09:30:43.773010 | 000c29b1-9f06-28d0-4abe-00000000c74d | SUMMARY | overcloud-controller-1 | Wait for puppet host configuration to finish | 133.30s 2024-01-17 09:30:43.773675 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-2 | Run puppet on the host to apply IPtables rules | 81.78s 2024-01-17 09:30:43.774343 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-0 | Run puppet on the host to apply IPtables rules | 80.99s 2024-01-17 09:30:43.774928 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-1 | Run puppet on the host to apply IPtables rules | 80.71s 2024-01-17 09:30:43.775545 | 000c29b1-9f06-28d0-4abe-00000000b7d8 | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 80.65s 2024-01-17 09:30:43.776155 | 000c29b1-9f06-28d0-4abe-00000000b8fe | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 80.21s 2024-01-17 09:30:43.776781 | 000c29b1-9f06-28d0-4abe-00000000b85e | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 76.84s 2024-01-17 09:30:43.777443 | 000c29b1-9f06-28d0-4abe-00000000b58a | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 66.95s 2024-01-17 09:30:43.778043 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-2 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.81s 2024-01-17 09:30:43.778660 | 000c29b1-9f06-28d0-4abe-00000000c7ab | SUMMARY | overcloud-novacompute-2 | Wait for puppet host configuration to finish | 42.09s 2024-01-17 09:30:43.779285 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-1 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.00s 2024-01-17 09:30:43.779870 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-0 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 40.08s 2024-01-17 09:30:43.780470 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781183 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781794 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.782388 | The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2 2024-01-17 09:30:43.783022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:45.104 477248 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo5qf1b220 ] cleaned up 2024-01-17 09:30:45.104 477248 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:45.105 477248 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2024-01-17 09:30:45.793 477248 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2024-01-17 09:30:45.863 477248 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud working_dir=self.working_dir) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1322, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 'BlacklistedHostnames', self.working_dir)) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 405, in config_download 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=forks 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2024-01-17 09:30:47.967 477248 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.971 477248 INFO osc_lib.shell [-] END return value: 1 If I list logs on the failing container and open the /etc/config.pp file I can see that bind_port, log_file, show_image_direct_url and show_multiple_locations are successfully read but it seems they are not defined for specified class. [root@overcloud-controller-0 ~]# podman logs f319ca15fd60 include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:15 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:15 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:15 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:15 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 09:19:13 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 09:19:13 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 09:19:13 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 09:19:13 puppet-user: has no parameter named 'bind_port' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'log_file' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x Also the content of config.pp is as follows: include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Does anyone know what is the issue here?