<div dir="ltr">Hi John,<div>Thanks for the inputs. Now I see something strange. </div><div>Deployment with external ceph is unstable, it got deployed once and we saw an error of VM not getting created because of some reasons, we were debugging and found that we found some NTP related observation, which we fixed and tried redeploying. </div><div>Not it again got failed at step 4:</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="box-sizing:border-box;color:rgb(36,36,36);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Web",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-25 17:34:29.036371 | 5254004d-021e-d4db-067d-000000007b1a |<span style="box-sizing:border-box"> </span>TASK | Create identity internal endpoint</span><br style="box-sizing:border-box;color:rgb(36,36,36);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Web",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><span style="box-sizing:border-box;color:rgb(36,36,36);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Web",sans-serif;font-size:14px;background-color:rgb(232,235,250)">2022-08-25 17:34:31.176105 | 5254004d-021e-d4db-067d-000000007b1a |<span style="box-sizing:border-box"> </span>FATAL | Create identity internal endpoint | undercloud | error={"changed": false, "extra_data": {"data": null, "details": "The request you have made requires authentication.", "response": "{\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}<span style="box-sizing:border-box">}</span>\n"}, "msg": "Failed to list services: Client Error for url: <a aria-label="Link https://overcloud-public.myhsc.com:13000/v3/services" title="https://overcloud-public.myhsc.com:13000/v3/services" href="https://overcloud-public.myhsc.com:13000/v3/services" rel="noopener noreferrer" target="_blank" tabindex="-1" style="box-sizing:border-box;background-color:transparent;outline-style:none;color:rgb(79,82,178);text-decoration-line:none">https://overcloud-public.mydomain.com:13000/v3/services</a>, The request you have made requires authentication."}</span><br></div><div><span style="box-sizing:border-box;color:rgb(36,36,36);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Web",sans-serif;font-size:14px;background-color:rgb(232,235,250)"><br></span></div></blockquote><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">To revalidate the case, I tried a fresh setup and saw that deployment again failed at step 4. </span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">and when we remove external ceph from the deployment command, we see that deployment is happening 100%. </span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">I see authorization errors, which I used to get earlier as well, but because of DNS we were able to resolve this. </span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">what could be the reason for this when we are using External ceph ?</span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">any inputs would be helpful</span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font color="#242424" face="-apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Apple Color Emoji, Segoe UI Emoji, Segoe UI Web, sans-serif"><span style="font-size:14px">deploy command:</span></font></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>stack@undercloud ~]$ cat deploy_step2.sh<br>openstack overcloud deploy --templates \<br> -r /home/stack/templates/roles_data.yaml \<br> -n /home/stack/templates/custom_network_data.yaml \<br> -e /home/stack/templates/overcloud-baremetal-deployed.yaml \<br> -e /home/stack/templates/networks-deployed-environment.yaml \<br> -e /home/stack/templates/vip-deployed-environment.yaml \<br> -e /home/stack/templates/environment.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml \<br> -e /home/stack/templates/ironic-config.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml \<br> -e /home/stack/templates/enable-tls.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml \<br> -e /home/stack/templates/cloudname.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor-hiera.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml \<br> -e /home/stack/templates/my-additional-ceph-settings.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \<br> -e /home/stack/containers-prepare-parameter.yaml<br></div><div><br></div><div><br></div></blockquote>[stack@undercloud ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml<br>resource_registry:<br> OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml<br><br>parameter_defaults:<br> # NOTE: These example parameters are required when using CephExternal<br> CephClusterFSID: 'ca3080e3-aa3a-4d1a-b1fd-483459a9ea4c'<br> CephClientKey: 'AQB2hMZi2u13NxAAVjmKopw+kNm6OnZOG7NktQ=='<br> CephExternalMonHost: 'abcd:abcd:abcd::11,abcd:abcd:abcd::12,abcd:abcd:abcd::13'<br><br> # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova<br> NovaEnableRbdBackend: true<br> CinderEnableRbdBackend: true<br> CinderBackupBackend: ceph<br> GlanceBackend: rbd<br> # Uncomment below if enabling legacy telemetry<br> # GnocchiBackend: rbd<br> # If the Ceph pools which host VMs, Volumes and Images do not match these<br> # names OR the client keyring to use is not named 'openstack', edit the<br> # following as needed.<br> NovaRbdPoolName: vms<br> CinderRbdPoolName: volumes<br> CinderBackupRbdPoolName: backups<br> GlanceRbdPoolName: images<br> # Uncomment below if enabling legacy telemetry<br> # GnocchiRbdPoolName: metrics<br> CephClientUserName: openstack<br><br> # finally we disable the Cinder LVM backend<br> CinderEnableIscsiBackend: false<br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 19, 2022 at 10:32 PM John Fulton <<a href="mailto:johfulto@redhat.com">johfulto@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Aug 19, 2022 at 3:45 AM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr">Hi Fulton,<div>Thanks for the inputs and apologies for the delay in response. </div><div>to my surprise passing the container prepare in standard worked for me, new container-prepare is:</div><div><br></div><div>parameter_defaults:<br> ContainerImagePrepare:<br> - push_destination: true<br> set:<br> ceph_alertmanager_image: alertmanager<br> ceph_alertmanager_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a><br> ceph_alertmanager_tag: v0.16.2<br> ceph_grafana_image: grafana<br> ceph_grafana_namespace: <a href="http://quay.ceph.io/app-sre" target="_blank">quay.ceph.io/app-sre</a><br> ceph_grafana_tag: 6.7.4<br> ceph_image: daemon<br> ceph_namespace: <a href="http://quay.io/ceph" target="_blank">quay.io/ceph</a><br> ceph_node_exporter_image: node-exporter<br> ceph_node_exporter_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a><br> ceph_node_exporter_tag: v0.17.0<br> ceph_prometheus_image: prometheus<br> ceph_prometheus_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a><br> ceph_prometheus_tag: v2.7.2<br> ceph_tag: v6.0.7-stable-6.0-pacific-centos-stream8<br> name_prefix: openstack-<br> name_suffix: ''<br> namespace: <a href="http://myserver.com:5000/tripleowallaby" target="_blank">myserver.com:5000/tripleowallaby</a><br> neutron_driver: ovn<br> rhel_containers: false<br> tag: current-tripleo<br> tag_from_label: rdo_version<br></div><div><br></div><div>But if we see or look at these containers I do not see any such containers available. we have tried looking at Undercloud and overcloud. </div></div></blockquote><div><br></div><div>The undercloud can download continers from the sources above and then act as a container registry. It's described here:</div><div><br></div><div><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/container_image_prepare.html" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/container_image_prepare.html</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr"><div>Also, the deployment is done when we are passing this config.</div><div>Thanks once again. </div><div><br></div><div>Also, we need to understand some use cases of using the storage from this external ceph, which can work as the mount for the VM as direct or Shared storage. Any idea or available document which tells more about how to consume external Ceph in the existing triple Overcloud?</div></div></blockquote><div><br></div><div>Ceph can provide OpenStack Block, Object and File storage and TripleO supports a variety of integration options for them.</div><div><br></div><div>TripleO can deploy Ceph as part of the OpenStack overcloud:</div><div><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html</a><br></div><div><br></div><div>TripleO can also deploy an OpenStack overcloud which uses an existing external ceph cluster:</div><div><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/ceph_external.html" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/ceph_external.html</a><br></div><div><br></div><div>At the end of both of these documents you can expect Glance, Nova, and Cinder to use Ceph block storage (RBD). </div><div><br></div><div>You can also have OpenStack use Ceph object storage (RGW). When RGW is used, a command like "openstack container create foo" will create an object storage container (not to be confused with podman/docker) on CephRGW as if your overcloud were running OpenStack Swift. If you have TripleO deploy Ceph as part of the OpenStack overcloud, RGW will be deployed and configured for OpenStack object storage by default (in Wallaby+). </div><div><br></div><div>The OpenStack Manila service can use CephFS as one of its backends. TripleO can deploy that too as described here:</div><div><br></div><div><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deploy_manila.html" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deploy_manila.html</a><br></div><div><br></div><div> John</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr"><div>Do share in case you know any, please.</div><div><br></div><div>Thanks once again for the support, it was really helpful</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 11, 2022 at 9:59 PM John Fulton <<a href="mailto:johfulto@redhat.com" target="_blank">johfulto@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="auto">The ceph container should no longer be needed for external ceph configuration (since the move from ceph-ansible to cephadm) but if removing the ceph env files makes the error go away, then try adding it back and then following these steps to prepare the ceph container on your undercloud before deploying. <div dir="auto"><br></div><div dir="auto"><a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#container-options" target="_blank">https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#container-options</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 10, 2022, 11:48 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr">Hi Thanks, <div>for the inputs, we could see the miss, </div><div>now we have added the required miss :</div><div>"<span style="font-family:monospace">TripleO resource </span>OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml"</div><div><br></div><div>Now with this setting if we deploy the setup in wallaby, we are getting error as:</div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br>PLAY [External deployment step 1] **********************************************<br>2022-08-11 08:33:20.183104 | 525400d4-7124-4a42-664c-0000000000a8 | TASK | External deployment step 1<br>2022-08-11 08:33:20.211821 | 525400d4-7124-4a42-664c-0000000000a8 | OK | External deployment step 1 | undercloud -> localhost | result={<br> "changed": false,<br> "msg": "Use --start-at-task 'External deployment step 1' to resume from this task"<br>}<br>[WARNING]: ('undercloud -> localhost', '525400d4-7124-4a42-664c-0000000000a8')<br>missing from stats<br>2022-08-11 08:33:20.254775 | 525400d4-7124-4a42-664c-0000000000a9 | TIMING | include_tasks | undercloud | 0:05:01.151528 | 0.03s<br>2022-08-11 08:33:20.304290 | 730cacb3-fa5a-4dca-9730-9a8ce54fb5a3 | INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/external_deploy_steps_tasks_step1.yaml | undercloud<br>2022-08-11 08:33:20.322079 | 525400d4-7124-4a42-664c-0000000048d0 | TASK | Set some tripleo-ansible facts<br>2022-08-11 08:33:20.350423 | 525400d4-7124-4a42-664c-0000000048d0 | OK | Set some tripleo-ansible facts | undercloud<br>2022-08-11 08:33:20.351792 | 525400d4-7124-4a42-664c-0000000048d0 | TIMING | Set some tripleo-ansible facts | undercloud | 0:05:01.248558 | 0.03s<br>2022-08-11 08:33:20.366717 | 525400d4-7124-4a42-664c-0000000048d7 | TASK | Container image prepare<br>2022-08-11 08:34:32.486108 | 525400d4-7124-4a42-664c-0000000048d7 | FATAL | Container image prepare | <b>undercloud | error={"changed": false, "error": "None: Max retries exceeded with url: /v2/ (Caused by None)", "msg": "Error running container image prepare: None: Max retries exceeded with url: /v2/ (Caused by None)", "params": {}, "success": false}</b><br>2022-08-11 08:34:32.488845 | 525400d4-7124-4a42-664c-0000000048d7 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:06:13.385607 | 72.12s<br><br></div></blockquote>This gets failed at step 1, As this is wallaby and based on the document (<a href="https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/ceph_external.html" rel="noreferrer" target="_blank">Use an external Ceph cluster with the Overcloud — TripleO 3.0.0 documentation (openstack.org)</a>) we should only pass this external-ceph.yaml for the external ceph intergration. </div><div>But it is not happening.</div><div><br></div><div><br></div><div>Few things to note:</div><div>1. Container Prepare:</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>(undercloud) [stack@undercloud ~]$ cat containers-prepare-parameter.yaml</div><div># Generated with the following on 2022-06-28T18:56:38.642315</div><div>#</div><div># openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml</div><div>#</div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> ContainerImagePrepare:</div><div> - push_destination: true</div><div> set:</div><div> name_prefix: openstack-</div><div> name_suffix: ''</div><div> namespace: <a href="http://myserver.com:5000/tripleowallaby" rel="noreferrer" target="_blank">myserver.com:5000/tripleowallaby</a></div><div> neutron_driver: ovn</div><div> rhel_containers: false</div><div> tag: current-tripleo</div><div> tag_from_label: rdo_version</div><div>(undercloud) [stack@undercloud ~]$</div></blockquote>2. this is SSL based deployment. <div><br></div><div>Any idea for the error, the issue is seen only once we have the external ceph integration enabled. </div><div><br></div><div>Best Regards,</div><div>Lokendra</div><div><br><div><div><br></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 4, 2022 at 7:22 PM Francesco Pantano <<a href="mailto:fpantano@redhat.com" rel="noreferrer" target="_blank">fpantano@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr"><div style="font-family:monospace">Hi,<br>ceph is supposed to be configured by this tripleo-ansible role [1], which is triggered by tht on external_deploy_steps [2].<br>In theory adding [3] should just work, assuming you customize the ceph cluster mon ip addresses, fsid and a few other related variables.<br>From your previous email I suspect in your external-ceph.yaml you missed the TripleO resource <span style="font-family:Arial,Helvetica,sans-serif">OS::TripleO::Services::CephExternal</span><span style="font-family:Arial,Helvetica,sans-serif">: </span><span style="font-family:Arial,Helvetica,sans-serif">../deployment/cephadm/ceph-client.yaml</span></div><div>(see [3]).<br><br>Thanks,<br>Francesco</div><div style="font-family:monospace"><br></div><div style="font-family:monospace"><br></div><div style="font-family:monospace">[1] <a href="https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo_ceph_client" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo_ceph_client</a></div><div style="font-family:monospace">[2] <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L93" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L93</a></div><div style="font-family:monospace">[3] <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/environments/external-ceph.yaml" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/master/environments/external-ceph.yaml</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 4, 2022 at 2:01 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" rel="noreferrer" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(77,77,77);padding-left:1ex"><div dir="ltr">Hi Team,<div>I was trying to integrate External Ceph with Triple0 Wallaby, and at the end of deployment in step4 getting the below error:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:21,158 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.157962 | 525400fe-86b8-65d9-d100-0000000080d2 | TASK | Create containers from /var/lib/tripleo-config/container-startup-config/step_4</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:21,239 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.238718 | 69e98219-f748-4af7-a6d0-f8f73680ce9b | INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/create.yml | overcloud-controller-2</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:21,273 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.272340 | 525400fe-86b8-65d9-d100-0000000086d9 | TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:24,532 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.530812 | | WARNING | ERROR: Can't run container nova_libvirt_init_secret</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">stderr:</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:24,533 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.532811 | 525400fe-86b8-65d9-d100-0000000082ec | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-</span><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(102,75,0)">config/step_4 | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed containers: nova_libvirt_init_secret"}</span><br style="box-sizing:border-box;color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)"><span style="color:rgb(191,191,191);font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif;font-size:14px;background-color:rgb(15,23,61)">2022-08-03 18:37:44,282 p=507732 u</span><br></div></blockquote></div><div><br></div><div><b>external-ceph.conf:</b></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> # Enable use of RBD backend in nova-compute</div><div> NovaEnableRbdBackend: True</div><div> # Enable use of RBD backend in cinder-volume</div><div> CinderEnableRbdBackend: True</div><div> # Backend to use for cinder-backup</div><div> CinderBackupBackend: ceph</div><div> # Backend to use for glance</div><div> GlanceBackend: rbd</div><div> # Name of the Ceph pool hosting Nova ephemeral images</div><div> NovaRbdPoolName: vms</div><div> # Name of the Ceph pool hosting Cinder volumes</div><div> CinderRbdPoolName: volumes</div><div> # Name of the Ceph pool hosting Cinder backups</div><div> CinderBackupRbdPoolName: backups</div><div> # Name of the Ceph pool hosting Glance images</div><div> GlanceRbdPoolName: images</div><div> # Name of the user to authenticate with the external Ceph cluster</div><div> CephClientUserName: admin</div><div> # The cluster FSID</div><div> CephClusterFSID: 'ca3080-aaaa-4d1a-b1fd-4aaaa9a9ea4c'</div><div> # The CephX user auth key</div><div> CephClientKey: 'AQDgRjhiuLMnAxAAnYwgERERFy0lzH6ufSl70A=='</div><div> # The list of Ceph monitors</div><div> CephExternalMonHost: 'abcd:abcd:abcd::11,abcd:abcd:abcd::12,abcd:abcd:abcd::13'</div><div>~</div></blockquote><div><div><br></div><div>Have tried checking and validating the ceph client details and they seem to be correct, further digging the container log I could see something like this :</div><div><br></div><div>[root@overcloud-novacompute-0 containers]# tail -f nova_libvirt_init_secret.log<br>tail: cannot open 'nova_libvirt_init_secret.log' for reading: No such file or directory<br>tail: no files remaining<br>[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_libvirt_init_secret.log<br>2022-08-04T11:48:47.689898197+05:30 stdout F ------------------------------------------------<br>2022-08-04T11:48:47.690002011+05:30 stdout F Initializing virsh secrets for: ceph:admin<br>2022-08-04T11:48:47.690590594+05:30 stdout F Error: /etc/ceph/ceph.conf was not found<br>2022-08-04T11:48:47.690625088+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin<br>2022-08-04T16:20:29.643785538+05:30 stdout F ------------------------------------------------<br>2022-08-04T16:20:29.643785538+05:30 stdout F Initializing virsh secrets for: ceph:admin<br>2022-08-04T16:20:29.644785532+05:30 stdout F Error: /etc/ceph/ceph.conf was not found<br>2022-08-04T16:20:29.644785532+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin<br>^C<br>[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_compute_init_log.log<br></div><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><font face="monospace">Francesco Pantano<br>
GPG KEY: F41BD75C</font><br></font></span></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr">~ Lokendra<br></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div>