<div dir="auto">The ceph container should no longer be needed for external ceph configuration (since the move from ceph-ansible to cephadm) but if removing the ceph env files makes the error go away,  then try adding it back and then following these steps to prepare the ceph container on your undercloud before deploying. <div dir="auto"><br></div><div dir="auto"><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#container-options">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#container-options</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 10, 2022, 11:48 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Thanks, <div>for the inputs, we could see the miss, </div><div>now we have added the required miss :</div><div>"<span style="font-family:monospace">TripleO resource </span>OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml"</div><div><br></div><div>Now with this setting if we deploy the setup in wallaby, we are getting error as:</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><br>PLAY [External deployment step 1] **********************************************<br>2022-08-11 08:33:20.183104 | 525400d4-7124-4a42-664c-0000000000a8 |       TASK | External deployment step 1<br>2022-08-11 08:33:20.211821 | 525400d4-7124-4a42-664c-0000000000a8 |         OK | External deployment step 1 | undercloud -> localhost | result={<br>    "changed": false,<br>    "msg": "Use --start-at-task 'External deployment step 1' to resume from this task"<br>}<br>[WARNING]: ('undercloud -> localhost', '525400d4-7124-4a42-664c-0000000000a8')<br>missing from stats<br>2022-08-11 08:33:20.254775 | 525400d4-7124-4a42-664c-0000000000a9 |     TIMING | include_tasks | undercloud | 0:05:01.151528 | 0.03s<br>2022-08-11 08:33:20.304290 | 730cacb3-fa5a-4dca-9730-9a8ce54fb5a3 |   INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/external_deploy_steps_tasks_step1.yaml | undercloud<br>2022-08-11 08:33:20.322079 | 525400d4-7124-4a42-664c-0000000048d0 |       TASK | Set some tripleo-ansible facts<br>2022-08-11 08:33:20.350423 | 525400d4-7124-4a42-664c-0000000048d0 |         OK | Set some tripleo-ansible facts | undercloud<br>2022-08-11 08:33:20.351792 | 525400d4-7124-4a42-664c-0000000048d0 |     TIMING | Set some tripleo-ansible facts | undercloud | 0:05:01.248558 | 0.03s<br>2022-08-11 08:33:20.366717 | 525400d4-7124-4a42-664c-0000000048d7 |       TASK | Container image prepare<br>2022-08-11 08:34:32.486108 | 525400d4-7124-4a42-664c-0000000048d7 |      FATAL | Container image prepare | <b>undercloud | error={"changed": false, "error": "None: Max retries exceeded with url: /v2/ (Caused by None)", "msg": "Error running container image prepare: None: Max retries exceeded with url: /v2/ (Caused by None)", "params": {}, "success": false}</b><br>2022-08-11 08:34:32.488845 | 525400d4-7124-4a42-664c-0000000048d7 |     TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:06:13.385607 | 72.12s<br><br></div></blockquote>This gets failed at step 1, As this is wallaby and based on the document (<a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/ceph_external.html" target="_blank" rel="noreferrer">Use an external Ceph cluster with the Overcloud — TripleO 3.0.0 documentation (openstack.org)</a>) we should only pass this external-ceph.yaml for the external ceph intergration. </div><div>But it is not happening.</div><div><br></div><div><br></div><div>Few things to note:</div><div>1. Container Prepare:</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>(undercloud) [stack@undercloud ~]$ cat containers-prepare-parameter.yaml</div><div># Generated with the following on 2022-06-28T18:56:38.642315</div><div>#</div><div>#   openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml</div><div>#</div></blockquote><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>parameter_defaults:</div><div>  ContainerImagePrepare:</div><div>  - push_destination: true</div><div>    set:</div><div>      name_prefix: openstack-</div><div>      name_suffix: ''</div><div>      namespace: <a href="http://myserver.com:5000/tripleowallaby" target="_blank" rel="noreferrer">myserver.com:5000/tripleowallaby</a></div><div>      neutron_driver: ovn</div><div>      rhel_containers: false</div><div>      tag: current-tripleo</div><div>    tag_from_label: rdo_version</div><div>(undercloud) [stack@undercloud ~]$</div></blockquote>2. this is SSL based deployment. <div><br></div><div>Any idea for the error, the issue is seen only once we have the external ceph integration enabled. </div><div><br></div><div>Best Regards,</div><div>Lokendra</div><div><br><div><div><br></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 4, 2022 at 7:22 PM Francesco Pantano <<a href="mailto:fpantano@redhat.com" target="_blank" rel="noreferrer">fpantano@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace">Hi,<br>ceph is supposed to be configured by this tripleo-ansible role [1], which is triggered by tht on external_deploy_steps [2].<br>In theory adding [3] should just work, assuming you customize the ceph cluster mon ip addresses, fsid and a few other related variables.<br>From your previous email I suspect in your external-ceph.yaml you missed the TripleO resource <span style="font-family:Arial,Helvetica,sans-serif">OS::TripleO::Services::CephExternal</span><span style="font-family:Arial,Helvetica,sans-serif">: </span><span style="font-family:Arial,Helvetica,sans-serif">../deployment/cephadm/ceph-client.yaml</span></div><div class="gmail_default">(see [3]).<br><br>Thanks,<br>Francesco</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">[1] <a href="https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo_ceph_client" target="_blank" rel="noreferrer">https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo_ceph_client</a></div><div class="gmail_default" style="font-family:monospace">[2] <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L93" target="_blank" rel="noreferrer">https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L93</a></div><div class="gmail_default" style="font-family:monospace">[3] <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/environments/external-ceph.yaml" target="_blank" rel="noreferrer">https://github.com/openstack/tripleo-heat-templates/blob/master/environments/external-ceph.yaml</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 4, 2022 at 2:01 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank" rel="noreferrer">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Team,<div>I was trying to integrate External Ceph with Triple0 Wallaby, and at the end of deployment in step4 getting the below error:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:21,158 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.157962 | 525400fe-86b8-65d9-d100-0000000080d2 |       TASK | Create containers from /var/lib/tripleo-config/container-startup-config/step_4</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:21,239 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.238718 | 69e98219-f748-4af7-a6d0-f8f73680ce9b |   INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/create.yml | overcloud-controller-2</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:21,273 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.272340 | 525400fe-86b8-65d9-d100-0000000086d9 |       TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:24,532 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.530812 |                                      |    WARNING | ERROR: Can't run container nova_libvirt_init_secret</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">stderr:</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:24,533 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.532811 | 525400fe-86b8-65d9-d100-0000000082ec |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-</span><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(255,255,0)">config/step_4 | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed containers: nova_libvirt_init_secret"}</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="color:rgb(36,36,36);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-03 18:37:44,282 p=507732 u</span><br></div></blockquote></div><div><br></div><div><b>external-ceph.conf:</b></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div>  # Enable use of RBD backend in nova-compute</div><div>  NovaEnableRbdBackend: True</div><div>  # Enable use of RBD backend in cinder-volume</div><div>  CinderEnableRbdBackend: True</div><div>  # Backend to use for cinder-backup</div><div>  CinderBackupBackend: ceph</div><div>  # Backend to use for glance</div><div>  GlanceBackend: rbd</div><div>  # Name of the Ceph pool hosting Nova ephemeral images</div><div>  NovaRbdPoolName: vms</div><div>  # Name of the Ceph pool hosting Cinder volumes</div><div>  CinderRbdPoolName: volumes</div><div>  # Name of the Ceph pool hosting Cinder backups</div><div>  CinderBackupRbdPoolName: backups</div><div>  # Name of the Ceph pool hosting Glance images</div><div>  GlanceRbdPoolName: images</div><div>  # Name of the user to authenticate with the external Ceph cluster</div><div>  CephClientUserName: admin</div><div>  # The cluster FSID</div><div>  CephClusterFSID: 'ca3080-aaaa-4d1a-b1fd-4aaaa9a9ea4c'</div><div>  # The CephX user auth key</div><div>  CephClientKey: 'AQDgRjhiuLMnAxAAnYwgERERFy0lzH6ufSl70A=='</div><div>  # The list of Ceph monitors</div><div>  CephExternalMonHost: 'abcd:abcd:abcd::11,abcd:abcd:abcd::12,abcd:abcd:abcd::13'</div><div>~</div></blockquote><div><div><br></div><div>Have tried checking and validating the ceph client details and they seem to be correct, further digging the container log I could see something like this :</div><div><br></div><div>[root@overcloud-novacompute-0 containers]# tail -f nova_libvirt_init_secret.log<br>tail: cannot open 'nova_libvirt_init_secret.log' for reading: No such file or directory<br>tail: no files remaining<br>[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_libvirt_init_secret.log<br>2022-08-04T11:48:47.689898197+05:30 stdout F ------------------------------------------------<br>2022-08-04T11:48:47.690002011+05:30 stdout F Initializing virsh secrets for: ceph:admin<br>2022-08-04T11:48:47.690590594+05:30 stdout F Error: /etc/ceph/ceph.conf was not found<br>2022-08-04T11:48:47.690625088+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin<br>2022-08-04T16:20:29.643785538+05:30 stdout F ------------------------------------------------<br>2022-08-04T16:20:29.643785538+05:30 stdout F Initializing virsh secrets for: ceph:admin<br>2022-08-04T16:20:29.644785532+05:30 stdout F Error: /etc/ceph/ceph.conf was not found<br>2022-08-04T16:20:29.644785532+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin<br>^C<br>[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_compute_init_log.log<br></div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><font face="monospace">Francesco Pantano<br>
GPG KEY: F41BD75C</font><br></font></span></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div>
</blockquote></div>