On Tue, May 7, 2024 at 1:25 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:

Hi,
I was updating the ssl template and then ran openstack overcloud
deploy and am getting error messages as below in most of the nodes:
2024-05-07 19:04:10.564330 | 48d539a1-1679-6e66-8636-00000000f0ab |
FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.573348 | 48d539a1-1679-6e66-8636-00000000f0ab |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 |
0:24:39.725509 | 1.78s
2024-05-07 19:04:10.578977 | 48d539a1-1679-6e66-8636-00000000f0ab |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 |
0:24:39.731144 | 1.78s
2024-05-07 19:04:10.582725 | 48d539a1-1679-6e66-8636-00000000f0ca |
FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.585487 | 48d539a1-1679-6e66-8636-00000000f0ca |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 |
0:24:39.737672 | 1.14s
2024-05-07 19:04:10.589799 | 48d539a1-1679-6e66-8636-00000000f0ca |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 |
0:24:39.741977 | 1.14s
2024-05-07 19:04:10.592984 | 48d539a1-1679-6e66-8636-00000000f23e |
TIMING | Run lvmfilter role | overcloud-novacompute-2 |
0:24:39.745173 | 0.11s
2024-05-07 19:04:10.595783 | 48d539a1-1679-6e66-8636-00000000f0ca |
FATAL | Pre-fetch all the containers | overcloud-cephstorage-3 |
item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo |
error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":
"Failed to run ['/bin/podman', 'inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json'] ['inspect',
'9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c',
'--format', 'json']: time=\"2024-05-07T11:04:10Z\" level=warning
msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-07T11:04:10Z\"
level=warning msg=\"Failed to decode the keys
[\\\"network.network_backend\\\"] from
\\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected
end of JSON input\n", "prefetch_image":
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"}
2024-05-07 19:04:10.598356 | 48d539a1-1679-6e66-8636-00000000f0ca |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 |
0:24:39.750539 | 1.06s
2024-05-07 19:04:10.602665 | 48d539a1-1679-6e66-8636-00000000f0ca |
TIMING | Pre-fetch all the containers | overcloud-cephstorage-3 |
0:24:39.754836 | 1.06s

Whereas in controller nodes I am not getting any errors when doing a
podman inspect using the same id.

[root@overcloud-cephstorage-0 ~]# podman inspect
9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c
--format json
Error: unexpected end of JSON input

Where the same command in controller node gives:

[root@overcloud-controller-0 ~]# podman inspect
9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c
--format json
[
{
"Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c",
"Digest":
"sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf",
"RepoTags": [
"172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"
],
"RepoDigests": [
"172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f",
"172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf"
],
"Parent": "",
"Comment": "",
"Created": "2022-02-09T05:23:45.932319128Z",
"Config": {
......

I do not understand what is happening and why this issue is coming.

With regards,
Swogat Pradhan