overcloud deploy fails in step 1 | tripleo | wallaby
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help?
With regards, Swogat Pradhan
Hi, Attaching the stdout for reference.
With regards, Swogat Pradhan
On Thu, May 16, 2024 at 6:31 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help?
With regards, Swogat Pradhan
Can someone please help me with this?
With regards, Swogat Pradhan
On Thu, May 16, 2024 at 6:39 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, Attaching the stdout for reference.
With regards, Swogat Pradhan
On Thu, May 16, 2024 at 6:31 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help?
With regards, Swogat Pradhan
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-manila-share:latest2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 http://172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the KeyError seen above. Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the KeyError seen above. Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
Hi, Can someone please assist on this issue?
With regards, Swogat Pradhan
On Fri, May 24, 2024 at 9:05 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the KeyError seen above. Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
You are beating the dead horse here. TripleO and wallaby both are dead (specially tripleO)
On Sun, May 26, 2024 at 1:32 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, Can someone please assist on this issue?
With regards, Swogat Pradhan
On Fri, May 24, 2024 at 9:05 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok:
[gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3:
kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org
,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1:
configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok:
[gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3:
kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org
,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1:
auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the
KeyError seen above.
Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
Hi Swogat,
On Fri, May 24, 2024 at 9:06 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
The error is the same with or without debug so the patch is related, so it may help or redirect to the actual underlying issue.
As the step has not failed on all the hosts, it needs to be checked on the failing hosts to see what triggered this. So you can check for the podman version there , and the output of "sudo podman image inspect <image>". The issue happens when the inspect image command fails(rc != 0) for some reason.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the
KeyError seen above.
Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
Hi Swogat,
Are you able to ssh this controller node manually? Which version of RHEL you are using?
Regards, Jitendra Mhatre
On Mon, 27 May, 2024, 7:09 pm Yatin Karel, ykarel@redhat.com wrote:
Hi Swogat,
On Fri, May 24, 2024 at 9:06 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
The error is the same with or without debug so the patch is related, so it may help or redirect to the actual underlying issue.
As the step has not failed on all the hosts, it needs to be checked on the failing hosts to see what triggered this. So you can check for the podman version there , and the output of "sudo podman image inspect <image>". The issue happens when the inspect image command fails(rc != 0) for some reason.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan <
swogatpradhan22@gmail.com> wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 391, in <module>\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 387, in main\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 132, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 370, in run\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 332, in run_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py",
line 318, in manage_container\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1659, in execute\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1572, in make_started\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 1351, in different\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 692, in __init__\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 699, in defaultize\n File
"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py",
line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the
KeyError seen above.
Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
-- Thanks and Regards Yatin Karel
Hi Jitendra, Yes, i am able to ssh to the servers manually.
With regards, Swogat Pradhan
On Tue, May 28, 2024 at 10:19 AM Jitendra Mhatre jsmhatre@gmail.com wrote:
Hi Swogat,
Are you able to ssh this controller node manually? Which version of RHEL you are using?
Regards, Jitendra Mhatre
On Mon, 27 May, 2024, 7:09 pm Yatin Karel, ykarel@redhat.com wrote:
Hi Swogat,
On Fri, May 24, 2024 at 9:06 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi Yatin, I am not sure if the link you shared applies to the issue I am facing. But here is the full error that i am getting after adding -vvv to the openstack overcloud deploy command:
2024-05-24 10:44:35,984 p=991970 u=stack n=ansible | 2024-05-24 10:44:35.983894 | 525400ff-fd40-184f-1e57-000000010e7d | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={ "changed": false, "module_stderr": "OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 172.25.201.98 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 172.25.201.98 originally 172.25.201.98\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 990376\r\ndebug3: mux_client_request_session: session request sent\r\nTraceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_rskro2xe/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }
Please let me know if it makes any sense.
The error is the same with or without debug so the patch is related, so it may help or redirect to the actual underlying issue.
As the step has not failed on all the hosts, it needs to be checked on the failing hosts to see what triggered this. So you can check for the podman version there , and the output of "sudo podman image inspect <image>". The issue happens when the inspect image command fails(rc != 0) for some reason.
With regards, Swogat Pradhan
On Mon, May 20, 2024 at 10:48 AM Yatin Karel ykarel@redhat.com wrote:
Hi Swogat,
On Thu, May 16, 2024 at 6:32 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error:
s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File "<stdin>", line 102, in <module>\n File "<stdin>", line 94, in _ansiballz_main\n File "<stdin>", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 391, in <module>\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 387, in main\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 132, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 370, in run\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 337, in check_failures\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator\n yield fs.pop().result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n return self.__get_result()\n File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n raise self._exception\n File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 332, in run_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py", line 318, in manage_container\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1659, in execute\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1572, in make_started\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 1351, in different\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 692, in __init__\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 699, in defaultize\n File "/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I do not have any idea why this is caused and how to fix it. Can someone please help? ]
I neither know what triggered this, but I see a fix[1] handling the KeyError seen above. Can try applying that patch and see if that resolves it.
[1] https://github.com/containers/ansible-podman-collections/pull/458
With regards, Swogat Pradhan
-- Thanks and Regards Yatin Karel
-- Thanks and Regards Yatin Karel
participants (4)
-
Jitendra Mhatre
-
Satish Patel
-
Swogat Pradhan
-
Yatin Karel