Hi, Do you by any chance have any solution for this?
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:59 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, All across the cluster i have below version of podman:
Version: 3.4.1-dev API Version: 3.4.1-dev Go Version: go1.16.7 Built: Tue Oct 19 16:11:42 2021 OS/Arch: linux/amd64
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:53 PM Jiri Podivin jpodivin@redhat.com wrote:
Hi, Swogat, Which version of Podman are you using?
On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan