Hello. I have a problem deploying ceph cluster prior to deploying OS with the following command. openstack overcloud ceph deploy -o /home/stack/3XOenvironment/deployed-ceph.yaml --stack overcloud --roles-data /home/stack/3XOenvironment/30-roles_data.yaml --network-data /home/stack/3XOenvironment/40-network_data.yaml --config /home/stack/3XOenvironment/ceph.conf --ntp-server 10.100.250.101 --container-image-prepare /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml -vvv I have successfully provisioned overcloud networks, vips and nodes (HPE GEN 9 machines). Ceph clusters seems to create but it fails on creating the environment file (deployed-ceph.yaml) for the openstack overcloud deploy command. The error involves tld_option which I cannot find anywhere. 2024-01-16 11:29:06.616835 | 000c29b1-9f06-81af-0614-00000000011b | TASK | Ceph cluster status 2024-01-16 11:29:06.654033 | 000c29b1-9f06-81af-0614-00000000011b | OK | Ceph cluster status | overcloud-controller-0 | result={ "changed": false, "msg": [ " cluster:", " id: 472da67d-c46c-5717-880e-574b9105d723", " health: HEALTH_OK", " ", " services:", " mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-2,overcloud-controller-1 (age 25m)", " mgr: overcloud-controller-0.ewcaqz(active, since 26m), standbys: overcloud-controller-2.gaxwws, overcloud-controller-1.muacne", " osd: 3 osds: 3 up (since 23m), 3 in (since 24m)", " ", " data:", " pools: 1 pools, 1 pgs", " objects: 0 objects, 0 B", " usage: 14 MiB used, 22 TiB / 22 TiB avail", " pgs: 1 active+clean", " " ] } 2024-01-16 11:29:06.655731 | 000c29b1-9f06-81af-0614-00000000011b | TIMING | tripleo_cephadm : Ceph cluster status | overcloud-controller-0 | 0:00:45.445615 | 0.04s 2024-01-16 11:29:06.668350 | 000c29b1-9f06-81af-0614-000000000131 | TASK | Create Deployed Ceph Heat Environment file to pass to overcloud deploy <localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack <localhost> EXEC /bin/sh -c 'echo ~stack && sleep 0' <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/stack/.ansible/tmp `"&& mkdir "` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" && echo ansible-tmp-1705400946.7091644-595040-96765848827570="` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" ) && sleep 0' <localhost> EXEC /bin/sh -c 'rm -f -r /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570/ > /dev/null 2>&1 && sleep 0' 2024-01-16 11:29:06.768346 | 000c29b1-9f06-81af-0614-000000000131 | FATAL | Create Deployed Ceph Heat Environment file to pass to overcloud deploy | overcloud-controller-0 -> localhost | error={ "changed": false, "msg": "AnsibleUndefinedVariable: {{ tld_option }}: 'tld_option' is undefined" } [WARNING]: ('overcloud-controller-0 -> localhost', '000c29b1-9f06-81af-0614-000000000131') missing from stats PLAY RECAP ********************************************************************* overcloud-controller-0 : ok=37 changed=8 unreachable=0 failed=1 skipped=43 rescued=0 ignored=1 undercloud : ok=10 changed=3 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024-01-16 11:29:06.777893 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.778573 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.779338 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:45.569227 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.780170 | UUID | Info | Host | Task Name | Run Time 2024-01-16 11:29:06.781078 | 000c29b1-9f06-81af-0614-000000000187 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public and private keys | 7.36s 2024-01-16 11:29:06.781746 | 000c29b1-9f06-81af-0614-000000000188 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public keys | 6.19s 2024-01-16 11:29:06.782621 | 000c29b1-9f06-81af-0614-00000000009b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set public/cluster network and v4/v6 ms_bind unless already in ceph.conf | 4.41s 2024-01-16 11:29:06.783807 | 000c29b1-9f06-81af-0614-0000000000b2 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 3.20s 2024-01-16 11:29:06.784945 | 000c29b1-9f06-81af-0614-00000000002d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 3.19s 2024-01-16 11:29:06.785972 | 000c29b1-9f06-81af-0614-0000000002fb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 1.27s 2024-01-16 11:29:06.786740 | 000c29b1-9f06-81af-0614-000000000118 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show the deployed daemons | 1.21s 2024-01-16 11:29:06.787585 | 000c29b1-9f06-81af-0614-00000000011a | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Print the status of the deployed Ceph cluster | 1.21s 2024-01-16 11:29:06.788369 | 000c29b1-9f06-81af-0614-000000000115 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.16s 2024-01-16 11:29:06.789142 | 000c29b1-9f06-81af-0614-00000000029f | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 1.12s 2024-01-16 11:29:06.789892 | 000c29b1-9f06-81af-0614-0000000000af | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.11s 2024-01-16 11:29:06.790586 | 000c29b1-9f06-81af-0614-00000000003b | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.89s 2024-01-16 11:29:06.791367 | 000c29b1-9f06-81af-0614-000000000184 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_admin_limit_file | 0.81s 2024-01-16 11:29:06.792182 | 000c29b1-9f06-81af-0614-000000000030 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 0.69s 2024-01-16 11:29:06.792987 | 000c29b1-9f06-81af-0614-000000000019 | SUMMARY | undercloud | Create Ceph spec based on baremetal_deployed_path and tripleo_roles | 0.68s 2024-01-16 11:29:06.793962 | 000c29b1-9f06-81af-0614-00000000003e | SUMMARY | overcloud-controller-0 | push it to tripleo_cephadm_assimilate_conf on bootstrap node | 0.66s 2024-01-16 11:29:06.794719 | 000c29b1-9f06-81af-0614-000000000185 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_non_admin_limit_file | 0.59s 2024-01-16 11:29:06.795418 | 000c29b1-9f06-81af-0614-00000000006e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.56s 2024-01-16 11:29:06.796092 | 000c29b1-9f06-81af-0614-00000000002e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.37s 2024-01-16 11:29:06.796735 | 000c29b1-9f06-81af-0614-000000000038 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.37s 2024-01-16 11:29:06.797373 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798652 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.799383 | The following node(s) had failures: overcloud-controller-0 2024-01-16 11:29:06.800015 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Temporary directory [ /tmp/tripleozr0mivzo ] cleaned up Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Temporary directory [ /tmp/tripleo9zw51ssj ] cleaned up Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_ceph.py", line 739, in take_action skip_tags=','.join(skip_tags), File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook raise RuntimeError(err_msg) RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 clean_up OvercloudCephDeploy: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 END return value: 1