Hi Thanks, 

for the inputs, we could see the miss, 

now we have added the required miss :

"TripleO resource OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml"

Now with this setting if we deploy the setup in wallaby, we are getting error as:

PLAY [External deployment step 1] **********************************************
2022-08-11 08:33:20.183104 | 525400d4-7124-4a42-664c-0000000000a8 |       TASK | External deployment step 1
2022-08-11 08:33:20.211821 | 525400d4-7124-4a42-664c-0000000000a8 |         OK | External deployment step 1 | undercloud -> localhost | result={
    "changed": false,
    "msg": "Use --start-at-task 'External deployment step 1' to resume from this task"
}
[WARNING]: ('undercloud -> localhost', '525400d4-7124-4a42-664c-0000000000a8')
missing from stats
2022-08-11 08:33:20.254775 | 525400d4-7124-4a42-664c-0000000000a9 |     TIMING | include_tasks | undercloud | 0:05:01.151528 | 0.03s
2022-08-11 08:33:20.304290 | 730cacb3-fa5a-4dca-9730-9a8ce54fb5a3 |   INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/external_deploy_steps_tasks_step1.yaml | undercloud
2022-08-11 08:33:20.322079 | 525400d4-7124-4a42-664c-0000000048d0 |       TASK | Set some tripleo-ansible facts
2022-08-11 08:33:20.350423 | 525400d4-7124-4a42-664c-0000000048d0 |         OK | Set some tripleo-ansible facts | undercloud
2022-08-11 08:33:20.351792 | 525400d4-7124-4a42-664c-0000000048d0 |     TIMING | Set some tripleo-ansible facts | undercloud | 0:05:01.248558 | 0.03s
2022-08-11 08:33:20.366717 | 525400d4-7124-4a42-664c-0000000048d7 |       TASK | Container image prepare
2022-08-11 08:34:32.486108 | 525400d4-7124-4a42-664c-0000000048d7 |      FATAL | Container image prepare | undercloud | error={"changed": false, "error": "None: Max retries exceeded with url: /v2/ (Caused by None)", "msg": "Error running container image prepare: None: Max retries exceeded with url: /v2/ (Caused by None)", "params": {}, "success": false}
2022-08-11 08:34:32.488845 | 525400d4-7124-4a42-664c-0000000048d7 |     TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:06:13.385607 | 72.12s

This gets failed at step 1, As this is wallaby and based on the document (Use an external Ceph cluster with the Overcloud — TripleO 3.0.0 documentation (openstack.org)) we should only pass this external-ceph.yaml for the external ceph intergration. 

But it is not happening.

Few things to note:

1. Container Prepare:

(undercloud) [stack@undercloud ~]$ cat containers-prepare-parameter.yaml

# Generated with the following on 2022-06-28T18:56:38.642315

#

#   openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml

#

parameter_defaults:

  ContainerImagePrepare:

  - push_destination: true

    set:

      name_prefix: openstack-

      name_suffix: ''

      namespace: myserver.com:5000/tripleowallaby

      neutron_driver: ovn

      rhel_containers: false

      tag: current-tripleo

    tag_from_label: rdo_version

(undercloud) [stack@undercloud ~]$

2. this is SSL based deployment. 

Any idea for the error, the issue is seen only once we have the external ceph integration enabled. 

Best Regards,

Lokendra

On Thu, Aug 4, 2022 at 7:22 PM Francesco Pantano <fpantano@redhat.com> wrote:

Hi,
ceph is supposed to be configured by this tripleo-ansible role [1], which is triggered by tht on external_deploy_steps [2].
In theory adding [3] should just work, assuming you customize the ceph cluster mon ip addresses, fsid and a few other related variables.
From your previous email I suspect in your external-ceph.yaml you missed the TripleO resource OS::TripleO::Services::CephExternal: ../deployment/cephadm/ceph-client.yaml

(see [3]).

Thanks,
Francesco

[1] https://github.com/openstack/tripleo-ansible/tree/master/tripleo_ansible/roles/tripleo_ceph_client

[2] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L93

[3] https://github.com/openstack/tripleo-heat-templates/blob/master/environments/external-ceph.yaml

On Thu, Aug 4, 2022 at 2:01 PM Lokendra Rathour <lokendrarathour@gmail.com> wrote:

Hi Team,

I was trying to integrate External Ceph with Triple0 Wallaby, and at the end of deployment in step4 getting the below error:

2022-08-03 18:37:21,158 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.157962 | 525400fe-86b8-65d9-d100-0000000080d2 |       TASK | Create containers from /var/lib/tripleo-config/container-startup-config/step_4
2022-08-03 18:37:21,239 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.238718 | 69e98219-f748-4af7-a6d0-f8f73680ce9b |   INCLUDED | /usr/share/ansible/roles/tripleo_container_manage/tasks/create.yml | overcloud-controller-2
2022-08-03 18:37:21,273 p=507732 u=stack n=ansible | 2022-08-03 18:37:21.272340 | 525400fe-86b8-65d9-d100-0000000086d9 |       TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4
2022-08-03 18:37:24,532 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.530812 |                                      |    WARNING | ERROR: Can't run container nova_libvirt_init_secret
stderr:
2022-08-03 18:37:24,533 p=507732 u=stack n=ansible | 2022-08-03 18:37:24.532811 | 525400fe-86b8-65d9-d100-0000000082ec |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed containers: nova_libvirt_init_secret"}
2022-08-03 18:37:44,282 p=507732 u

external-ceph.conf:

parameter_defaults:

  # Enable use of RBD backend in nova-compute

  NovaEnableRbdBackend: True

  # Enable use of RBD backend in cinder-volume

  CinderEnableRbdBackend: True

  # Backend to use for cinder-backup

  CinderBackupBackend: ceph

  # Backend to use for glance

  GlanceBackend: rbd

  # Name of the Ceph pool hosting Nova ephemeral images

  NovaRbdPoolName: vms

  # Name of the Ceph pool hosting Cinder volumes

  CinderRbdPoolName: volumes

  # Name of the Ceph pool hosting Cinder backups

  CinderBackupRbdPoolName: backups

  # Name of the Ceph pool hosting Glance images

  GlanceRbdPoolName: images

  # Name of the user to authenticate with the external Ceph cluster

  CephClientUserName: admin

  # The cluster FSID

  CephClusterFSID: 'ca3080-aaaa-4d1a-b1fd-4aaaa9a9ea4c'

  # The CephX user auth key

  CephClientKey: 'AQDgRjhiuLMnAxAAnYwgERERFy0lzH6ufSl70A=='

  # The list of Ceph monitors

  CephExternalMonHost: 'abcd:abcd:abcd::11,abcd:abcd:abcd::12,abcd:abcd:abcd::13'

~

Have tried checking and validating the ceph client details and they seem to be correct, further digging the container log I could see something like this :

[root@overcloud-novacompute-0 containers]# tail -f nova_libvirt_init_secret.log
tail: cannot open 'nova_libvirt_init_secret.log' for reading: No such file or directory
tail: no files remaining
[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_libvirt_init_secret.log
2022-08-04T11:48:47.689898197+05:30 stdout F ------------------------------------------------
2022-08-04T11:48:47.690002011+05:30 stdout F Initializing virsh secrets for: ceph:admin
2022-08-04T11:48:47.690590594+05:30 stdout F Error: /etc/ceph/ceph.conf was not found
2022-08-04T11:48:47.690625088+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin
2022-08-04T16:20:29.643785538+05:30 stdout F ------------------------------------------------
2022-08-04T16:20:29.643785538+05:30 stdout F Initializing virsh secrets for: ceph:admin
2022-08-04T16:20:29.644785532+05:30 stdout F Error: /etc/ceph/ceph.conf was not found
2022-08-04T16:20:29.644785532+05:30 stdout F Path to nova_libvirt_init_secret was ceph:admin
^C
[root@overcloud-novacompute-0 containers]# tail -f stdouts/nova_compute_init_log.log

--

~ Lokendra

skype: lokendrarathour

--

Francesco Pantano
GPG KEY: F41BD75C

--

~ Lokendra

skype: lokendrarathour