FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | item={'service_name': 'cinderv3', 'service_type': 'volume'} | error={"ansible_index_var": "cinder_api_service", "ansible_loop_var": "item", "changed": false, "cinder_api_service": 1, "item": {"service_name": "cinderv3", "service_type": "volume"}, "module_stderr": "Failed to discover available identity versions when contacting https://overcloud.bdxworld.com:13000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 677, in urlopen\n chunked=chunked,\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 381, in make_request\n self._validate_conn(conn)\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 978, in _validate_conn\n conn.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 371, in connect\n ssl_context=context,\n File \"/usr/lib/python3.6/site-packages/urllib3/util/ssl.py\", line 384, in ssl_wrap_socket\n return context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\", line 365, in wrap_socket\n context=self, _session=session)\n File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init\n self.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n self._sslobj.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 648, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.bdxworld.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, *kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.bdxworld.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init_\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.bdxworld.com:13000: HTTPSConnectionPool(host='overcloud.bdxworld.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in \n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call_\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.bdxworld.com:13000: HTTPSConnectionPool(host='overcloud.bdxworld.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"\", line 102, in \n File \"\", line 94, in _ansiballz_main\n File \"\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in \n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_yfxm9irl/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 141, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plugin\n 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://overcloud.bdxworld.com:13000: HTTPSConnectionPool(host='overcloud.bdxworld.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} Catalog list: +-----------+----------------+--------------------------------------------------------------------------------------+ | Name | Type | Endpoints | +-----------+----------------+--------------------------------------------------------------------------------------+ | aodh | alarming | regionOne | | | | internal: http://172.25.201.250:8042 | | | | regionOne | | | | public: http://172.25.201.150:8042 | | | | regionOne | | | | admin: http://172.25.201.250:8042 | | | | | | placement | placement | regionOne | | | | admin: http://172.25.201.250:8778/placement | | | | regionOne | | | | internal: http://172.25.201.250:8778/placement | | | | regionOne | | | | public: http://172.25.201.150:8778/placement | | | | | | gnocchi | metric | regionOne | | | | public: http://172.25.201.150:8041 | | | | regionOne | | | | internal: http://172.25.201.250:8041 | | | | regionOne | | | | admin: http://172.25.201.250:8041 | | | | | | glance | image | regionOne | | | | internal: http://172.25.201.250:9292 | | | | regionOne | | | | admin: http://172.25.201.250:9292 | | | | regionOne | | | | public: http://172.25.201.150:9292 | | | | | | keystone | identity | regionOne | | | | internal: http://172.25.201.250:5000 | | | | regionOne | | | | admin: http://172.25.201.250:35357 | | | | regionOne | | | | public: https://overcloud.bdxworld.com:13000 | | | | | | heat-cfn | cloudformation | regionOne | | | | internal: http://172.25.201.250:8000/v1 | | | | regionOne | | | | public: http://172.25.201.150:8000/v1 | | | | regionOne | | | | admin: http://172.25.201.250:8000/v1 | | | | | | neutron | network | regionOne | | | | admin: http://172.25.201.250:9696 | | | | regionOne | | | | public: http://172.25.201.150:9696 | | | | regionOne | | | | internal: http://172.25.201.250:9696 | | | | | | heat | orchestration | regionOne | | | | internal: http://172.25.201.250:8004/v1/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | public: http://172.25.201.150:8004/v1/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.201.250:8004/v1/5d922243077045c48fe4b075e386551b | | | | | | octavia | load-balancer | regionOne | | | | public: http://172.25.201.150:9876 | | | | regionOne | | | | admin: http://172.25.201.250:9876 | | | | regionOne | | | | internal: http://172.25.201.250:9876 | | | | | | cinderv3 | volumev3 | regionOne | | | | internal: http://172.25.201.250:8776/v3/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | public: http://172.25.201.150:8776/v3/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.201.250:8776/v3/5d922243077045c48fe4b075e386551b | | | | | | swift | object-store | regionOne | | | | public: http://172.25.201.150:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.202.50:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | regionOne | | | | internal: http://172.25.202.50:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | | | nova | compute | regionOne | | | | admin: http://172.25.201.250:8774/v2.1 | | | | regionOne | | | | internal: http://172.25.201.250:8774/v2.1 | | | | regionOne | | | | public: http://172.25.201.150:8774/v2.1 | | | | Host entries: [stack@hkg2director ~]$ cat /etc/hosts # START_HOST_ENTRIES_FOR_STACK: overcloud 172.25.201.68 hkg2director.ctlplane.bdxworld.com hkg2director.ctlplane hkg2director.ctlplane.bdxcloud.bdxworld.com 172.25.201.91 overcloud.ctlplane.bdxworld.com 172.25.202.50 overcloud.storage.bdxworld.com 172.25.202.90 overcloud.storagemgmt.bdxworld.com 172.25.201.250 overcloud.internalapi.bdxworld.com 172.25.201.150 overcloud.bdxworld.com 172.25.201.212 overcloud-controller-0.bdxworld.com overcloud-controller-0 172.25.202.13 overcloud-controller-0.storage.bdxworld.com overcloud-controller-0.storage 172.25.202.75 overcloud-controller-0.storagemgmt.bdxworld.com overcloud-controller-0.storagemgmt 172.25.201.212 overcloud-controller-0.internalapi.bdxworld.com overcloud-controller-0.internalapi 172.25.202.143 overcloud-controller-0.tenant.bdxworld.com overcloud-controller-0.tenant 172.25.201.144 overcloud-controller-0.external.bdxworld.com overcloud-controller-0.external 172.25.201.106 overcloud-controller-0.ctlplane.bdxworld.com overcloud-controller-0.ctlplane 172.25.201.205 overcloud-controller-1.bdxworld.com overcloud-controller-1 172.25.202.18 overcloud-controller-1.storage.bdxworld.com overcloud-controller-1.storage 172.25.202.76 overcloud-controller-1.storagemgmt.bdxworld.com overcloud-controller-1.storagemgmt 172.25.201.205 overcloud-controller-1.internalapi.bdxworld.com overcloud-controller-1.internalapi 172.25.202.142 overcloud-controller-1.tenant.bdxworld.com overcloud-controller-1.tenant 172.25.201.149 overcloud-controller-1.external.bdxworld.com overcloud-controller-1.external 172.25.201.105 overcloud-controller-1.ctlplane.bdxworld.com overcloud-controller-1.ctlplane 172.25.201.201 overcloud-controller-2.bdxworld.com overcloud-controller-2 172.25.202.12 overcloud-controller-2.storage.bdxworld.com overcloud-controller-2.storage 172.25.202.74 overcloud-controller-2.storagemgmt.bdxworld.com overcloud-controller-2.storagemgmt 172.25.201.201 overcloud-controller-2.internalapi.bdxworld.com overcloud-controller-2.internalapi 172.25.202.149 overcloud-controller-2.tenant.bdxworld.com overcloud-controller-2.tenant 172.25.201.139 overcloud-controller-2.external.bdxworld.com overcloud-controller-2.external 172.25.201.97 overcloud-controller-2.ctlplane.bdxworld.com overcloud-controller-2.ctlplane 172.25.201.209 overcloud-controller-no-ceph-3.bdxworld.com overcloud-controller-no-ceph-3 172.25.202.17 overcloud-controller-no-ceph-3.storage.bdxworld.com overcloud-controller-no-ceph-3.storage 172.25.202.79 overcloud-controller-no-ceph-3.storagemgmt.bdxworld.com overcloud-controller-no-ceph-3.storagemgmt 172.25.201.209 overcloud-controller-no-ceph-3.internalapi.bdxworld.com overcloud-controller-no-ceph-3.internalapi 172.25.202.135 overcloud-controller-no-ceph-3.tenant.bdxworld.com overcloud-controller-no-ceph-3.tenant 172.25.201.137 overcloud-controller-no-ceph-3.external.bdxworld.com overcloud-controller-no-ceph-3.external 172.25.201.103 overcloud-controller-no-ceph-3.ctlplane.bdxworld.com overcloud-controller-no-ceph-3.ctlplane 172.25.201.202 overcloud-novacompute-0.bdxworld.com overcloud-novacompute-0 172.25.202.19 overcloud-novacompute-0.storage.bdxworld.com overcloud-novacompute-0.storage 172.25.201.202 overcloud-novacompute-0.internalapi.bdxworld.com overcloud-novacompute-0.internalapi 172.25.202.140 overcloud-novacompute-0.tenant.bdxworld.com overcloud-novacompute-0.tenant 172.25.201.107 overcloud-novacompute-0.ctlplane.bdxworld.com overcloud-novacompute-0.ctlplane 172.25.201.207 overcloud-novacompute-1.bdxworld.com overcloud-novacompute-1 172.25.202.15 overcloud-novacompute-1.storage.bdxworld.com overcloud-novacompute-1.storage 172.25.201.207 overcloud-novacompute-1.internalapi.bdxworld.com overcloud-novacompute-1.internalapi 172.25.202.144 overcloud-novacompute-1.tenant.bdxworld.com overcloud-novacompute-1.tenant 172.25.201.112 overcloud-novacompute-1.ctlplane.bdxworld.com overcloud-novacompute-1.ctlplane 172.25.201.200 overcloud-novacompute-2.bdxworld.com overcloud-novacompute-2 172.25.202.20 overcloud-novacompute-2.storage.bdxworld.com overcloud-novacompute-2.storage 172.25.201.200 overcloud-novacompute-2.internalapi.bdxworld.com overcloud-novacompute-2.internalapi 172.25.202.138 overcloud-novacompute-2.tenant.bdxworld.com overcloud-novacompute-2.tenant 172.25.201.100 overcloud-novacompute-2.ctlplane.bdxworld.com overcloud-novacompute-2.ctlplane 172.25.201.199 overcloud-novacompute-3.bdxworld.com overcloud-novacompute-3 172.25.202.9 overcloud-novacompute-3.storage.bdxworld.com overcloud-novacompute-3.storage 172.25.201.199 overcloud-novacompute-3.internalapi.bdxworld.com overcloud-novacompute-3.internalapi 172.25.202.141 overcloud-novacompute-3.tenant.bdxworld.com overcloud-novacompute-3.tenant 172.25.201.104 overcloud-novacompute-3.ctlplane.bdxworld.com overcloud-novacompute-3.ctlplane 172.25.201.198 overcloud-novacompute-4.bdxworld.com overcloud-novacompute-4 172.25.202.16 overcloud-novacompute-4.storage.bdxworld.com overcloud-novacompute-4.storage 172.25.201.198 overcloud-novacompute-4.internalapi.bdxworld.com overcloud-novacompute-4.internalapi 172.25.202.139 overcloud-novacompute-4.tenant.bdxworld.com overcloud-novacompute-4.tenant 172.25.201.109 overcloud-novacompute-4.ctlplane.bdxworld.com overcloud-novacompute-4.ctlplane 172.25.202.14 overcloud-cephstorage-0.bdxworld.com overcloud-cephstorage-0 172.25.202.14 overcloud-cephstorage-0.storage.bdxworld.com overcloud-cephstorage-0.storage 172.25.202.72 overcloud-cephstorage-0.storagemgmt.bdxworld.com overcloud-cephstorage-0.storagemgmt 172.25.201.101 overcloud-cephstorage-0.ctlplane.bdxworld.com overcloud-cephstorage-0.ctlplane 172.25.202.6 overcloud-cephstorage-1.bdxworld.com overcloud-cephstorage-1 172.25.202.6 overcloud-cephstorage-1.storage.bdxworld.com overcloud-cephstorage-1.storage 172.25.202.78 overcloud-cephstorage-1.storagemgmt.bdxworld.com overcloud-cephstorage-1.storagemgmt 172.25.201.92 overcloud-cephstorage-1.ctlplane.bdxworld.com overcloud-cephstorage-1.ctlplane 172.25.202.11 overcloud-cephstorage-2.bdxworld.com overcloud-cephstorage-2 172.25.202.11 overcloud-cephstorage-2.storage.bdxworld.com overcloud-cephstorage-2.storage 172.25.202.85 overcloud-cephstorage-2.storagemgmt.bdxworld.com overcloud-cephstorage-2.storagemgmt 172.25.201.111 overcloud-cephstorage-2.ctlplane.bdxworld.com overcloud-cephstorage-2.ctlplane 172.25.202.10 overcloud-cephstorage-3.bdxworld.com overcloud-cephstorage-3 172.25.202.10 overcloud-cephstorage-3.storage.bdxworld.com overcloud-cephstorage-3.storage 172.25.202.87 overcloud-cephstorage-3.storagemgmt.bdxworld.com overcloud-cephstorage-3.storagemgmt 172.25.201.96 overcloud-cephstorage-3.ctlplane.bdxworld.com overcloud-cephstorage-3.ctlplane 172.25.201.68 hkg2director.ctlplane.bdxworld.com hkg2director.ctlplane hkg2director.ctlplane.bdxcloud.bdxworld.com # END_HOST_ENTRIES_FOR_STACK: overcloud # START_HOST_ENTRIES_FOR_STACK: undercloud 172.25.201.68 hkg2director.bdxworld.com hkg2director 172.25.201.68 hkg2director.external.bdxworld.com hkg2director.external 172.25.201.68 hkg2director.ctlplane.bdxworld.com hkg2director.ctlplane 172.25.201.68 bdxworld.com 172.25.201.68 bdxworld.com # END_HOST_ENTRIES_FOR_STACK: undercloud 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.25.201.68 hkg2director.bdxworld.com hkg2director bdxworld.com