Hi Team,

we were trying to deploy OpenStack wallaby, and we see that 4 out of 5 runs deployment is getting failed for mentioned below reasons:

Error:

2022-08-25 17:34:29.036371 | 5254004d-021e-d4db-067d-000000007b1a |       TASK | Create identity internal endpoint
2022-08-25 17:34:31.176105 | 5254004d-021e-d4db-067d-000000007b1a |      FATAL | Create identity internal endpoint | undercloud | error={"changed": false, "extra_data": {"data": null, "details": "The request you have made requires authentication.", "response": "{\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"}, "msg": "Failed to list services: Client Error for url:  overcloud-public.myhsc.com :13000/v3/services, The request you have made requires authentication."}

Debug logs:

pending results....
The full traceback is:
  File "/tmp/ansible_openstack.cloud.endpoint_payload_qhlqb_qw/ansible_openstack.cloud.endpoint_payload.zip/ansible_collections/open          stack/cloud/plugins/module_utils/openstack.py", line 407, in __call__
    results = self.run()
  File "/tmp/ansible_openstack.cloud.endpoint_payload_qhlqb_qw/ansible_openstack.cloud.endpoint_payload.zip/ansible_collections/open          stack/cloud/plugins/modules/endpoint.py", line 150, in run
  File "/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 537, in get_service
    return _utils._get_entity(self, 'service', name_or_id, filters)
  File "/usr/lib/python3.6/site-packages/openstack/cloud/_utils.py", line 197, in _get_entity
    entities = search(name_or_id, filters, **kwargs)
  File "/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 517, in search_services
    services = self.list_services()
  File "/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 501, in list_services
    error_message="Failed to list services")
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 395, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 668, in request
    return _json_response(response, error_message=error_message)
  File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 646, in _json_response
    exceptions.raise_from_response(response, error_message=error_message)
  File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 238, in raise_from_response
    http_status=http_status, request_id=request_id
2022-08-25 19:38:03.201899 | 5254004d-021e-7578-cd65-000000007ad6 |      FATAL | Create identity internal endpoint | undercloud | er          ror={
    "changed": false,
    "extra_data": {
        "data": null,
        "details": "The request you have made requires authentication.",
        "response": "{\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unautho          rized\"}}\n"
    },
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "availability_zone": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "enabled": true,
            "endpoint_interface": "internal",
            "interface": "public",
            "region": "regionOne",
            "region_name": null,
            "service": "keystone",
            "state": "present",
            "timeout": 180,
            "url": "http://[fd00:fd00:fd00:2000::368]:5000",
            "validate_certs": null,
            "wait": true
        }
    },
    "msg": "Failed to list services: Client Error for url: https://overcloud-public.myhsc.com:13000/v3/services, The request you have made requires authentication."
}

DeployCommand that was used:

 openstack overcloud deploy --templates \
    -r /home/stack/templates/roles_data.yaml \
    -n /home/stack/templates/custom_network_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor-hiera.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
    -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
    -e /home/stack/templates/networks-deployed-environment.yaml \
    -e /home/stack/templates/vip-deployed-environment.yaml \
    -e /home/stack/templates/environment.yaml \
    -e /home/stack/templates/ironic-config.yaml \
    -e /home/stack/templates/enable-tls.yaml \
    -e /home/stack/templates/cloudname.yaml \
    -e /home/stack/templates/my-additional-ceph-settings.yaml \
    -e /home/stack/containers-prepare-parameter.yaml

[Issue]

we are seeing the deployment is working perfectly once and the same setting is not working perfectly in the second run. 

the failure rate is high. 

what can be the reasons behind this? 

Note:

Before this, we were deploying with DNS and SSL and it was perfectly working fine in multiple reruns.

But after SSL we have seen this random failure. 

--

~ Lokendra

skype: lokendrarathour