<div dir="ltr"><div>Hello,</div><div><br></div><div>After digging after grafana, it seems it needed to download something from the internet, and i didn't really configure a proper gateway on the external network.</div><div>So I started by configuring a proper gateway and I tested it with the half deployed nodes, the I redid the deployment, and again I got this error :</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>2021-08-24 21:29:29.616805 | 525400e8-92c8-d397-6f7e-000000006133 | FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | error={"changed": false, "module_stderr": "Fa<br>iled to discover available identity versions when contacting <a href="http://10.0.2.40:5000">http://10.0.2.40:5000</a>. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/si<br>te-packages/urllib3/connection.py\", line 162, in _new_conn\n (self._dns_host, self.port), self.timeout, **extra_kw)\n File \"/usr/lib/python3.6/site-packages/urllib3/util/connection.py<br>\", line 80, in create_connection\n raise err\n File \"/usr/lib/python3.6/site-packages/urllib3/util/connection.py\", line 70, in create_connection\n sock.connect(sa)\nTimeoutError: <br>[Errno 110] Connection timed out\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urll<br>ib3/connectionpool.py\", line 600, in urlopen\n chunked=chunked)\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 354, in _make_request\n conn.request(meth<br>od, url, **httplib_request_kw)\n File \"/usr/lib64/python3.6/http/client.py\", line 1269, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/lib6<br>4/python3.6/http/client.py\", line 1315, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1264, in endheaders<br>\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib64/pyt<br>hon3.6/http/client.py\", line 978, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 184, in connect\n conn = self._new_conn()\n File <br>\"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 171, in _new_conn\n self, \"Failed to establish a new connection: %s\" % e)\nurllib3.exceptions.NewConnectionError: <urll<br>ib3.connection.HTTPConnection object at 0x7f96f7b10cc0>: Failed to establish a new connection: [Errno 110] Connection timed out\n\nDuring handling of the above exception, another exception <br>occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-p<br>ackages/urllib3/connectionpool.py\", line 638, in urlopen\n _stacktrace=sys.exc_info()[2])\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 399, in increment\n <br> raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.2.40', port=5000): Max retries exceeded with url: / (Caused<br>by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f96f7b10cc0>: Failed to establish a new connection: [Errno 110] Connection timed out',))\n\nDuring handling of the ab$<br>ve exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 997, in _send_request\n resp $<br> self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File \"/u$<br>r/lib/python3.6/site-packages/requests/sessions.py\", line 646, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 516$<br> in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='10.0.2.40', port=5000): Max retries exceeded with url: / (Caused by N$wConnectionError('<urllib3.connection.HTTPConnection object at 0x7f96f7b10cc0>: Failed to establish a new connection: [Errno 110] Connection timed out',))\n\nDuring handling of the above e$<br>ception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plug$<br>n\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/pyt$<br>on3.6/site-packages/keystoneauth1/discover.py\", line 1442, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keys$<br>oneauth1/discover.py\", line 526, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 101, in get_version_data\n r$<br>sp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1116, in get\n return self.request(url, '$<br>ET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 906, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneaut$<br>1/session.py\", line 1013, in _send_request\n raise exceptions.ConnectFailure(msg)\nkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to <a href="http://10.0.2.4">http://10.0.2.4</a>$<br>:5000: HTTPConnectionPool(host='10.0.2.40', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f96f7b10cc0>: Failed<br>to establish a new connection: [Errno 110] Connection timed out',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<$<br>tdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_m$<br>dule\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_$<br>ame)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_service_payload_wcyk6h37/ansible_os_keystone_service_p$<br>yload.zip/ansible/modules/cloud/openstack/os_keystone_service.py\", line 194, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_wcyk6h37/ansible_os_keystone_service_payload.zi$<br>/ansible/modules/cloud/openstack/os_keystone_service.py\", line 153, in main\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 510, in search_services\n se$<br>vices = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 485, in list_services\n if self._is_client_version('identity', 2):\n File \$<br>/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 459, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/op$<br>nstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.$<br>y\", line 406, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 282, in get_endpoint\n return <a href="http://self.se">self.se</a>$<br>sion.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1218, in get_endpoint\n return auth.get_endpoint(self, **kwarg$<br>)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/$<br>ite-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/key$<br>toneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", l$<br>ne 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plu$<br>in\n 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that $our auth_url is correct. <b>Unable to establish connection to <a href="http://10.0.2.40:5000">http://10.0.2.40:5000</a>: HTTPConnectionPool(host='10.0.2.40', port=5000): Max retries exceeded with url: / (Caused by NewConnectionE<br>rror('<urllib3.connection.HTTPConnection object at 0x7f96f7b10cc0>: Failed to establish a new connection: [Errno 110] Connection timed out',))\n", "module_stdout": "", "msg": "MODULE FAILUR<br>E\nSee stdout/stderr for the exact error", "rc": 1} </b> <br>2021-08-24 21:29:29.617697 | 525400e8-92c8-d397-6f7e-000000006133 | TIMING | Clean up legacy Cinder keystone catalog entries | undercloud | 1:07:40.666419 | 130.85s <br> <br>PLAY RECAP ********************************************************************* <br>overcloud-computehci-0 : ok=260 changed=145 unreachable=0 failed=0 skipped=140 rescued=0 ignored=0 <br>overcloud-computehci-1 : ok=258 changed=145 unreachable=0 failed=0 skipped=140 rescued=0 ignored=0 <br>overcloud-computehci-2 : ok=255 changed=145 unreachable=0 failed=0 skipped=140 rescued=0 ignored=0 <br>overcloud-controller-0 : ok=295 changed=181 unreachable=0 failed=0 skipped=151 rescued=0 ignored=0 <br>overcloud-controller-1 : ok=289 changed=177 unreachable=0 failed=0 skipped=152 rescued=0 ignored=0 <br>overcloud-controller-2 : ok=288 changed=177 unreachable=0 failed=0 skipped=152 rescued=0 ignored=0 <br>undercloud : ok=105 changed=21 unreachable=0 failed=1 skipped=45 rescued=0 ignored=0 <br> <br>2021-08-24 21:29:29.730778 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <br>2021-08-24 21:29:29.731007 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1723 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <br>2021-08-24 21:29:29.731098 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 1:07:40.779840 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <br>2021-08-24 21:29:29.731172 | UUID | Info | Host | Task Name | Run Time <br>2021-08-24 21:29:29.731251 | 525400e8-92c8-d397-6f7e-000000003b9a | SUMMARY | undercloud | Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log | 1762.93s <br>2021-08-24 21:29:29.731349 | 525400e8-92c8-d397-6f7e-0000000057aa | SUMMARY | undercloud | tripleo-ceph-run-ansible : run ceph-ansible | 990.24s <br>2021-08-24 21:29:29.731433 | 525400e8-92c8-d397-6f7e-000000005951 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 133.22s <br>2021-08-24 21:29:29.731503 | 525400e8-92c8-d397-6f7e-000000006133 | SUMMARY | undercloud | Clean up legacy Cinder keystone catalog entries | 130.85s <br>2021-08-24 21:29:29.731569 | 525400e8-92c8-d397-6f7e-000000006012 | SUMMARY | overcloud-controller-0 | Wait for containers to start for step 3 using paunch | 103.45s <br>2021-08-24 21:29:29.731643 | 525400e8-92c8-d397-6f7e-000000004337 | SUMMARY | overcloud-computehci-0 | Pre-fetch all the containers | 94.00s <br>2021-08-24 21:29:29.731729 | 525400e8-92c8-d397-6f7e-000000004378 | SUMMARY | overcloud-computehci-2 | Pre-fetch all the containers | 92.64s <br>2021-08-24 21:29:29.731795 | 525400e8-92c8-d397-6f7e-000000004337 | SUMMARY | overcloud-computehci-1 | Pre-fetch all the containers | 86.38s <br>2021-08-24 21:29:29.731867 | 525400e8-92c8-d397-6f7e-000000004d68 | SUMMARY | overcloud-controller-0 | Wait for container-puppet tasks (generate config) to finish | 84.13s <br>2021-08-24 21:29:29.731946 | 525400e8-92c8-d397-6f7e-000000004d99 | SUMMARY | overcloud-controller-2 | Wait for container-puppet tasks (generate config) to finish | 80.76s <br>2021-08-24 21:29:29.732012 | 525400e8-92c8-d397-6f7e-00000000427c | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 80.21s <br>2021-08-24 21:29:29.732073 | 525400e8-92c8-d397-6f7e-00000000427c | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 77.03s <br>2021-08-24 21:29:29.732138 | 525400e8-92c8-d397-6f7e-0000000042f5 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 76.32s <br>2021-08-24 21:29:29.732202 | 525400e8-92c8-d397-6f7e-000000004dd3 | SUMMARY | overcloud-controller-1 | Wait for container-puppet tasks (generate config) to finish | 74.36s <br>2021-08-24 21:29:29.732266 | 525400e8-92c8-d397-6f7e-000000005da7 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for ovn_dbs | 68.39s <br>2021-08-24 21:29:29.732329 | 525400e8-92c8-d397-6f7e-000000005ce2 | SUMMARY | overcloud-controller-0 | Wait for containers to start for step 2 using paunch | 64.55s <br>2021-08-24 21:29:29.732398 | 525400e8-92c8-d397-6f7e-000000004b97 | SUMMARY | overcloud-controller-2 | Wait for puppet host configuration to finish | 58.13s <br>2021-08-24 21:29:29.732463 | 525400e8-92c8-d397-6f7e-000000004c1a | SUMMARY | overcloud-controller-1 | Wait for puppet host configuration to finish | 58.11s <br>2021-08-24 21:29:29.732526 | 525400e8-92c8-d397-6f7e-000000005bd3 | SUMMARY | overcloud-controller-1 | Wait for containers to start for step 2 using paunch | 58.09s <br>2021-08-24 21:29:29.732589 | 525400e8-92c8-d397-6f7e-000000005b9b | SUMMARY | overcloud-controller-2 | Wait for containers to start for step 2 using paunch | 58.09s </div></blockquote><div><br></div><div><br></div><div>Thank you again for your assistance.<br></div><div><br></div><div>Regards.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 24 août 2021 à 08:59, wodel youchi <<a href="mailto:wodel.youchi@gmail.com">wodel.youchi@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, and thanks for your help</div><div><br></div><div>As for Ceph, here is container prepare</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">parameter_defaults:
</span><br> ContainerImagePrepare:
<br> - push_destination: true
<br> set:
<br> ceph_alertmanager_image: alertmanager
<br> ceph_alertmanager_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a>
<br> ceph_alertmanager_tag: v0.16.2
<br> ceph_grafana_image: grafana
<br> ceph_grafana_namespace: <a href="http://quay.ceph.io/app-sre" target="_blank">quay.ceph.io/app-sre</a>
<br> <b>ceph_grafana_tag: 5.4.3</b>
<br> ceph_image: daemon
<br> ceph_namespace: <a href="http://quay.ceph.io/ceph-ci" target="_blank">quay.ceph.io/ceph-ci</a>
<br> ceph_node_exporter_image: node-exporter
<br> ceph_node_exporter_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a>
<br> ceph_node_exporter_tag: v0.17.0
<br> ceph_prometheus_image: prometheus
<br> ceph_prometheus_namespace: <a href="http://quay.ceph.io/prometheus" target="_blank">quay.ceph.io/prometheus</a>
<br> ceph_prometheus_tag: v2.7.2
<br> <b>ceph_tag: v4.0.19-stable-4.0-nautilus-centos-7-x86_64</b>
<br> name_prefix: centos-binary-
<br> name_suffix: ''
<br> namespace: <a href="http://quay.io/tripleotraincentos8" target="_blank">quay.io/tripleotraincentos8</a>
<br> neutron_driver: ovn
<br> rhel_containers: false
<br> tag: current-tripleo
<br> tag_from_label: rdo_version</span></div><div><span style="font-family:monospace"><br></span></div><div>And yes, the <a href="http://10.200.7.0/24" target="_blank">10.200.7.0/24</a> network is my storage network</div><div>Here is a snippet from my network_data.yaml</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(175,95,0);background-color:rgb(255,255,255)">- </span><span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">name</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> Storage
</span><br> <span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">vip</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">true</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
</span><br> <span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">vlan</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">1107</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
</span><br> <span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">name_lower</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> storage
</span><br> <span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">ip_subnet</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">'<a href="http://10.200.7.0/24" target="_blank">10.200.7.0/24</a>'</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
</span><br> <span style="color:rgb(24,178,178);background-color:rgb(255,255,255)">allocation_pools</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">[{</span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">'start'</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">'10.200.7.150'</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">, </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">'end'</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">:</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">'10.200.7.169'</span><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)">}]</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)"><br></span></span></div><div><span style="font-family:monospace"><span style="color:rgb(178,24,178);background-color:rgb(255,255,255)"></span></span></div><div>I will look into the grafana service to see why it's not booting and get back to you.<br></div><div><br></div><div>Regards.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 23 août 2021 à 17:28, Francesco Pantano <<a href="mailto:fpantano@redhat.com" target="_blank">fpantano@redhat.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace">Hello,</div><div class="gmail_default" style="font-family:monospace">thanks John for your reply here.<br>A few more comments inline:</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 23, 2021 at 6:16 PM John Fulton <<a href="mailto:johfulto@redhat.com" target="_blank">johfulto@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Aug 23, 2021 at 10:52 AM wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I redid the undercloud deployment for the Train version for now. And I verified the download URL for the images.<br>
> My overcloud deployment has moved forward but I still get errors.<br>
><br>
> This is what I got this time :<br>
>><br>
>> "TASK [ceph-grafana : wait for grafana to start] ********************************",<br>
>> "Monday 23 August 2021 14:55:21 +0100 (0:00:00.961) 0:12:59.319 ********* ",<br>
>> "fatal: [overcloud-controller-0]: FAILED! => {\"changed\": false, \"elapsed\": 300, \"msg\": \"Timeout when waiting for 10.20<br>
>> 0.7.151:3100\"}",<br>
>> "fatal: [overcloud-controller-1]: FAILED! => {\"changed\": false, \"elapsed\": 300, \"msg\": \"Timeout when waiting for 10.20<br>
>> 0.7.155:3100\"}",<br>
>> "fatal: [overcloud-controller-2]: FAILED! => {\"changed\": false, \"elapsed\": 300, \"msg\": \"<span class="gmail_default" style="font-family:monospace"></span>Timeout when waiting for 10.20<br>
>> 0.7.165:3100\"}",<br>
<br>
I'm not certain of the ceph-ansible version you're using but it should<br>
be a version 4 with train. ceph-ansible should already be installed on<br>
your undercloud judging by this error and in the latest version 4 this<br>
task is where it failed:<br>
<br>
<a href="https://github.com/ceph/ceph-ansible/blob/v4.0.64/roles/ceph-grafana/tasks/configure_grafana.yml#L112-L115" rel="noreferrer" target="_blank">https://github.com/ceph/ceph-ansible/blob/v4.0.64/roles/ceph-grafana/tasks/configure_grafana.yml#L112-L115</a><br>
<br>
You can check the status of this service on your three controllers and<br>
then debug it directly.</blockquote><div><span class="gmail_default" style="font-family:monospace">As John pointed out, ceph-ansible is able to configure, render and start the associated<br>systemd unit for all the ceph monitoring stack components (node-exported, prometheus, alertmanager and<br>grafana).<br>You can ssh to your controllers, and check the systemd unit associated, checking the journal to see why<br>they failed to start (I saw there's a timeout waiting for the container to start).<br>A potential plan, in this case, could be:<br><br>1. check the systemd unit (I guess you can start with grafana which is the failed service)<br>2. look at the journal logs (feel free to attach here the relevant part of the output)</span></div><div><span class="gmail_default" style="font-family:monospace">3. double check the network where the service is bound (can you attach the /var/lib/mistral/<stack>/ceph-ansible/group_vars/all.yaml)<br> * The grafana process should be run on the storage network, but I see a "</span><span class="gmail_default" style="font-family:monospace"></span>Timeout when waiting for <a href="http://10.200.7.165:3100" target="_blank">10.200.7.165:3100</a><span class="gmail_default" style="font-family:monospace">": is that network the right one?</span></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
John<br>
<br>
>> "RUNNING HANDLER [ceph-prometheus : service handler] ****************************",<br>
>> "Monday 23 August 2021 15:00:22 +0100 (0:05:00.767) 0:18:00.087 ********* ",<br>
>> "PLAY RECAP *********************************************************************",<br>
>> "overcloud-computehci-0 : ok=224 changed=23 unreachable=0 failed=0 skipped=415 rescued=0 ignored=0 ",<br>
>> "overcloud-computehci-1 : ok=199 changed=18 unreachable=0 failed=0 skipped=392 rescued=0 ignored=0 ",<br>
>> "overcloud-computehci-2 : ok=212 changed=23 unreachable=0 failed=0 skipped=390 rescued=0 ignored=0 ",<br>
>> "overcloud-controller-0 : ok=370 changed=52 unreachable=0 failed=1 skipped=539 rescued=0 ignored=0 ",<br>
>> "overcloud-controller-1 : ok=308 changed=43 unreachable=0 failed=1 skipped=495 rescued=0 ignored=0 ",<br>
>> "overcloud-controller-2 : ok=317 changed=45 unreachable=0 failed=1 skipped=493 rescued=0 ignored=0 ",<br>
>><br>
>> "INSTALLER STATUS ***************************************************************",<br>
>> "Install Ceph Monitor : Complete (0:00:52)",<br>
>> "Install Ceph Manager : Complete (0:05:49)",<br>
>> "Install Ceph OSD : Complete (0:02:28)",<br>
>> "Install Ceph RGW : Complete (0:00:27)",<br>
>> "Install Ceph Client : Complete (0:00:33)",<br>
>> "Install Ceph Grafana : In Progress (0:05:54)",<br>
>> "\tThis phase can be restarted by running: roles/ceph-grafana/tasks/main.yml",<br>
>> "Install Ceph Node Exporter : Complete (0:00:28)",<br>
>> "Monday 23 August 2021 15:00:22 +0100 (0:00:00.006) 0:18:00.094 ********* ",<br>
>> "=============================================================================== ",<br>
>> "ceph-grafana : wait for grafana to start ------------------------------ 300.77s",<br>
>> "ceph-facts : get ceph current status ---------------------------------- 300.27s",<br>
>> "ceph-container-common : pulling <a href="http://udtrain.ctlplane.umaitek.dz:8787/ceph-ci/daemon:v4.0.19-stable-4.0-nautilus-centos-7-x86_64" rel="noreferrer" target="_blank">udtrain.ctlplane.umaitek.dz:8787/ceph-ci/daemon:v4.0.19-stable-4.0-nautilus-centos-7-x86_64</a><br>
>> image -- 19.04s",<br>
>> "ceph-mon : waiting for the monitor(s) to form the quorum... ------------ 12.83s",<br>
>> "ceph-osd : use ceph-volume lvm batch to create bluestore osds ---------- 12.13s",<br>
>> "ceph-osd : wait for all osd to be up ----------------------------------- 11.88s",<br>
>> "ceph-osd : set pg_autoscale_mode value on pool(s) ---------------------- 11.00s",<br>
>> "ceph-osd : create openstack pool(s) ------------------------------------ 10.80s",<br>
>> "ceph-grafana : make sure grafana is down ------------------------------- 10.66s",<br>
>> "ceph-osd : customize pool crush_rule ----------------------------------- 10.15s",<br>
>> "ceph-osd : customize pool size ----------------------------------------- 10.15s",<br>
>> "ceph-osd : customize pool min_size ------------------------------------- 10.14s",<br>
>> "ceph-osd : assign application to pool(s) ------------------------------- 10.13s",<br>
>> "ceph-osd : list existing pool(s) ---------------------------------------- 8.59s",<br>
>><br>
>> "ceph-mon : fetch ceph initial keys -------------------------------------- 7.01s",<br>
>> "ceph-container-common : get ceph version -------------------------------- 6.75s",<br>
>> "ceph-prometheus : start prometheus services ----------------------------- 6.67s",<br>
>> "ceph-mgr : wait for all mgr to be up ------------------------------------ 6.66s",<br>
>> "ceph-grafana : start the grafana-server service ------------------------- 6.33s",<br>
>> "ceph-mgr : create ceph mgr keyring(s) on a mon node --------------------- 6.26s"<br>
>> ],<br>
>> "failed_when_result": true<br>
>> }<br>
>> 2021-08-23 15:00:24.427687 | 525400e8-92c8-47b1-e162-00000000597d | TIMING | tripleo-ceph-run-ansible : print ceph-ansible outpu$<br>
>> in case of failure | undercloud | 0:37:30.226345 | 0.25s<br>
>><br>
>> PLAY RECAP *********************************************************************<br>
>> overcloud-computehci-0 : ok=213 changed=117 unreachable=0 failed=0 skipped=120 rescued=0 ignored=0<br>
>> overcloud-computehci-1 : ok=207 changed=117 unreachable=0 failed=0 skipped=120 rescued=0 ignored=0<br>
>> overcloud-computehci-2 : ok=207 changed=117 unreachable=0 failed=0 skipped=120 rescued=0 ignored=0<br>
>> overcloud-controller-0 : ok=237 changed=145 unreachable=0 failed=0 skipped=128 rescued=0 ignored=0<br>
>> overcloud-controller-1 : ok=232 changed=145 unreachable=0 failed=0 skipped=128 rescued=0 ignored=0<br>
>> overcloud-controller-2 : ok=232 changed=145 unreachable=0 failed=0 skipped=128 rescued=0 ignored=0<br>
>> undercloud : ok=100 changed=18 unreachable=0 failed=1 skipped=37 rescued=0 ignored=0<br>
>><br>
>> 2021-08-23 15:00:24.559997 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.560328 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1366 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.560419 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:37:30.359090 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.560490 | UUID | Info | Host | Task Name | Run Time<br>
>> 2021-08-23 15:00:24.560589 | 525400e8-92c8-47b1-e162-00000000597b | SUMMARY | undercloud | tripleo-ceph-run-ansible : run ceph-ans<br>
>> ible | 1082.71s<br>
>> 2021-08-23 15:00:24.560675 | 525400e8-92c8-47b1-e162-000000004d9a | SUMMARY | overcloud-controller-1 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 356.02s<br>
>> 2021-08-23 15:00:24.560763 | 525400e8-92c8-47b1-e162-000000004d6a | SUMMARY | overcloud-controller-0 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 355.74s<br>
>> 2021-08-23 15:00:24.560839 | 525400e8-92c8-47b1-e162-000000004dd0 | SUMMARY | overcloud-controller-2 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 355.68s<br>
>> 2021-08-23 15:00:24.560912 | 525400e8-92c8-47b1-e162-000000003bb1 | SUMMARY | undercloud | Run tripleo-container-image-prepare log<br>
>> ged to: /var/log/tripleo-container-image-prepare.log | 143.03s<br>
>> 2021-08-23 15:00:24.560986 | 525400e8-92c8-47b1-e162-000000004b13 | SUMMARY | overcloud-controller-0 | Wait for puppet host config<br>
>> uration to finish | 125.36s<br>
>> 2021-08-23 15:00:24.561057 | 525400e8-92c8-47b1-e162-000000004b88 | SUMMARY | overcloud-controller-2 | Wait for puppet host config<br>
>> uration to finish | 125.33s<br>
>> 2021-08-23 15:00:24.561128 | 525400e8-92c8-47b1-e162-000000004b4b | SUMMARY | overcloud-controller-1 | Wait for puppet host config<br>
>> uration to finish | 125.25s<br>
>> 2021-08-23 15:00:24.561300 | 525400e8-92c8-47b1-e162-000000001dc4 | SUMMARY | overcloud-controller-2 | Run puppet on the host to a<br>
>> pply IPtables rules | 108.08s<br>
>> 2021-08-23 15:00:24.561374 | 525400e8-92c8-47b1-e162-000000001e4f | SUMMARY | overcloud-controller-0 | Run puppet on the host to a<br>
>> pply IPtables rules | 107.34s<br>
>> 2021-08-23 15:00:24.561444 | 525400e8-92c8-47b1-e162-000000004c8d | SUMMARY | overcloud-computehci-2 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 96.56s<br>
>> 2021-08-23 15:00:24.561514 | 525400e8-92c8-47b1-e162-000000004c33 | SUMMARY | overcloud-computehci-0 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 96.38s<br>
>> 2021-08-23 15:00:24.561580 | 525400e8-92c8-47b1-e162-000000004c60 | SUMMARY | overcloud-computehci-1 | Wait for container-puppet t<br>
>> asks (generate config) to finish | 93.41s<br>
>> 2021-08-23 15:00:24.561645 | 525400e8-92c8-47b1-e162-00000000434d | SUMMARY | overcloud-computehci-0 | Pre-fetch all the container<br>
>> s | 92.70s<br>
>> 2021-08-23 15:00:24.561712 | 525400e8-92c8-47b1-e162-0000000043ed | SUMMARY | overcloud-computehci-2 | Pre-fetch all the container<br>
>> s | 91.90s<br>
>> 2021-08-23 15:00:24.561782 | 525400e8-92c8-47b1-e162-000000004385 | SUMMARY | overcloud-computehci-1 | Pre-fetch all the container<br>
>> s | 91.88s<br>
>> 2021-08-23 15:00:24.561876 | 525400e8-92c8-47b1-e162-00000000491c | SUMMARY | overcloud-computehci-1 | Wait for puppet host config<br>
>> uration to finish | 90.37s<br>
>> 2021-08-23 15:00:24.561947 | 525400e8-92c8-47b1-e162-000000004951 | SUMMARY | overcloud-computehci-2 | Wait for puppet host config<br>
>> uration to finish | 90.37s<br>
>> 2021-08-23 15:00:24.562016 | 525400e8-92c8-47b1-e162-0000000048e6 | SUMMARY | overcloud-computehci-0 | Wait for puppet host config<br>
>> uration to finish | 90.35s<br>
>> 2021-08-23 15:00:24.562080 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.562196 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.562311 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~<br>
>> 2021-08-23 15:00:24.562379 | The following node(s) had failures: undercloud<br>
>> 2021-08-23 15:00:24.562456 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>> Host 10.0.2.40 not found in /home/stack/.ssh/known_hosts<br>
>> Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.Overcloud Endpoint: <a href="http://10.0.2.40:5000" rel="noreferrer" target="_blank">http://10.0.2.40:5000</a><br>
>> Overcloud Horizon Dashboard URL: <a href="http://10.0.2.40:80/dashboard" rel="noreferrer" target="_blank">http://10.0.2.40:80/dashboard</a><br>
>> Overcloud rc file: /home/stack/overcloudrc<br>
>> Overcloud Deployed with error<br>
>> Overcloud configuration failed.<br>
>><br>
><br>
><br>
> Could someone help debug this, the ansible.log is huge, I can't see what's the origin of the problem, if someone can point me to the right direction it will aprecciated.<br>
> Thanks in advance.<br>
><br>
> Regards.<br>
><br>
> Le mer. 18 août 2021 à 18:02, Wesley Hayutin <<a href="mailto:whayutin@redhat.com" target="_blank">whayutin@redhat.com</a>> a écrit :<br>
>><br>
>><br>
>><br>
>> On Wed, Aug 18, 2021 at 10:10 AM Dmitry Tantsur <<a href="mailto:dtantsur@redhat.com" target="_blank">dtantsur@redhat.com</a>> wrote:<br>
>>><br>
>>> Hi,<br>
>>><br>
>>> On Wed, Aug 18, 2021 at 4:39 PM wodel youchi <<a href="mailto:wodel.youchi@gmail.com" target="_blank">wodel.youchi@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Hi,<br>
>>>> I am trying to deploy openstack with tripleO using VMs and nested-KVM for the compute node. This is for test and learning purposes.<br>
>>>><br>
>>>> I am using the Train version and following some tutorials.<br>
>>>> I prepared my different template files and started the deployment, but I got these errors :<br>
>>>><br>
>>>> Failed to provision instance fc40457e-4b3c-4402-ae9d-c528f2c2ad30: Asynchronous exception: Node failed to deploy. Exception: Agent API for node 6d3724fc-6f13-4588-bbe5-56bc4f9a4f87 returned HTTP status code 404 with error: Not found: Extension with id iscsi not found. for node<br>
>>>><br>
>>><br>
>>> You somehow ended up using master (Xena release) deploy ramdisk with Train TripleO. You need to make sure to download Train images. I hope TripleO people can point you at the right place.<br>
>>><br>
>>> Dmitry<br>
>><br>
>><br>
>> <a href="http://images.rdoproject.org/centos8/" rel="noreferrer" target="_blank">http://images.rdoproject.org/centos8/</a><br>
>> <a href="http://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/" rel="noreferrer" target="_blank">http://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/</a><br>
>><br>
>>><br>
>>><br>
>>>><br>
>>>> and<br>
>>>><br>
>>>> Got HTTP 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'CUSTOM_BAREMETAL' on resource provider '6d3724fc-6f13-4588-bbe5-56bc4f9a4f87'. The requested amount would exceed the capacity. ",<br>
>>>><br>
>>>> Could you help understand what those errors mean? I couldn't find anything similar on the net.<br>
>>>><br>
>>>> Thanks in advance.<br>
>>>><br>
>>>> Regards.<br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Red Hat GmbH, <a href="https://de.redhat.com/" rel="noreferrer" target="_blank">https://de.redhat.com/</a> , Registered seat: Grasbrunn,<br>
>>> Commercial register: Amtsgericht Muenchen, HRB 153243,<br>
>>> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill<br>
<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><font face="monospace">Francesco Pantano<br>
GPG KEY: F41BD75C</font><br></font></span></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>