podman inspect : Error: unexpected end of JSON input | openstack wallaby | tripleo
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan
Hi, Swogat, Which version of Podman are you using?
On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from
\"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ " 172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ " 172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f ", " 172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf " ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan
Hi, All across the cluster i have below version of podman:
Version: 3.4.1-dev API Version: 3.4.1-dev Go Version: go1.16.7 Built: Tue Oct 19 16:11:42 2021 OS/Arch: linux/amd64
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:53 PM Jiri Podivin jpodivin@redhat.com wrote:
Hi, Swogat, Which version of Podman are you using?
On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan
Hi, Do you by any chance have any solution for this?
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:59 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, All across the cluster i have below version of podman:
Version: 3.4.1-dev API Version: 3.4.1-dev Go Version: go1.16.7 Built: Tue Oct 19 16:11:42 2021 OS/Arch: linux/amd64
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:53 PM Jiri Podivin jpodivin@redhat.com wrote:
Hi, Swogat, Which version of Podman are you using?
On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan
Hi All, I tried pulling the image manually and it succeeded using podman pull and I still cannot inspect it due to the error 'unexpected end of JSON input'. Do I need to delete the image and then retry pulling the image again? And if so then i guess the containers using the said images will stop and get killed, which will affect my infra.
With regards, Swogat Pradhan
On Wed, May 8, 2024 at 8:58 AM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, Do you by any chance have any solution for this?
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:59 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, All across the cluster i have below version of podman:
Version: 3.4.1-dev API Version: 3.4.1-dev Go Version: go1.16.7 Built: Tue Oct 19 16:11:42 2021 OS/Arch: linux/amd64
With regards, Swogat Pradhan
On Tue, May 7, 2024 at 5:53 PM Jiri Podivin jpodivin@redhat.com wrote:
Hi, Swogat, Which version of Podman are you using?
On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan swogatpradhan22@gmail.com wrote:
Hi, I was updating the ssl template and then ran openstack overcloud deploy and am getting error messages as below in most of the nodes: 2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.725509 | 1.78s 2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:24:39.731144 | 1.78s 2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.737672 | 1.14s 2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:24:39.741977 | 1.14s 2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e | TIMING | Run lvmfilter role | overcloud-novacompute-2 | 0:24:39.745173 | 0.11s 2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca | FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\ntime="2024-05-07T11:04:10Z" level=warning msg="Failed to decode the keys [\"network.network_backend\"] from \"/usr/share/containers/containers.conf\"."\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.750539 | 1.06s 2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca | TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 | 0:24:39.754836 | 1.06s
Whereas in controller nodes I am not getting any errors when doing a podman inspect using the same id.
[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json Error: unexpected end of JSON input
Where the same command in controller node gives:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c --format json [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", "Config": { ......
I do not understand what is happening and why this issue is coming.
With regards, Swogat Pradhan
participants (2)
-
Jiri Podivin
-
Swogat Pradhan