Hello, I am faced with an issue where I can't run kolla-ansible prechecks against one of the system and the only thing unique about this system is that its the one I used to bootstrap ceph, hence have cephadm installed. It makes me assume that the two might be incompatible. Ceph is using a system account called cephuser and kolla is using an system account called kolla, so I am lost why they are stumping on each other. Granted, both have full sudo, so that could be the route where this mess came up. The three systems are both execution machine and storage machine. When I initially set it up, I made assumptions that cinder was part of the core service. I then struggled to bring up the virtual machine and its while debugging this issue that I realized I need cinder. Fine, I enabled cinder on /etc/kolla/globals.d/cinder.yml as documented, but now I can't run prechecks. This was failing only on cinder. When I run deploy, now its failing on glance too. Have anyone noticed something similar? Can it be fixed without reinstalling all the systems? This is the error I am seeing:- The full traceback is: Traceback (most recent call last): File "/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py", line 107, in <module> _ansiballz_main() File "/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py", line 99, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py", line 47, in invoke_module runpy.run_module(mod_name='ansible.modules.kolla_container_facts', init_globals=dict(_module_fqn='ansible.modules.kolla_container_facts', _modlib_path=modlib_path), File "/usr/lib64/python3.9/runpy.py", line 225, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) File "/usr/lib64/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py", line 132, in <module> File "/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py", line 124, in main File "/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py", line 105, in use_podman File "/usr/local/lib/python3.9/site-packages/podman/domain/manager.py", line 68, in reload latest = self.manager.get(self.id) File "/usr/local/lib/python3.9/site-packages/podman/domain/containers_manager.py", line 44, in get response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/podman/api/client.py", line 75, in raise_for_status raise APIError(cause, response=self._response, explanation=message) podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (getting container from store "1127b70a453e15c7b5e2ba8da307f8b17791e6a23fb2b2e94a4d5dd5fce59907": container not known) fatal: [ostack-49.openstack.example.com]: FAILED! => { "changed": false, "module_stderr": "OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host ostack-49.openstack.example.com originally ostack-49.openstack.example.com\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug1: configuration requests final Match pass\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host ostack-49.openstack.example.com originally ostack-49.openstack.example.com\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/lib/kolla/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/lib/kolla/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 881450\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to ostack-49.openstack.example.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py\", line 107, in <module>\r\n _ansiballz_main()\r\n File \"/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py\", line 99, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/var/lib/kolla/.ansible/tmp/ansible-tmp-1724965733.8092885-882811-135535224643810/AnsiballZ_kolla_container_facts.py\", line 47, in invoke_module\r\n runpy.run_module(mod_name='ansible.modules.kolla_container_facts', init_globals=dict(_module_fqn='ansible.modules.kolla_container_facts', _modlib_path=modlib_path),\r\n File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\r\n return _run_module_code(code, init_globals, run_name, mod_spec)\r\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\r\n _run_code(code, mod_globals, init_globals,\r\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py\", line 132, in <module>\r\n File \"/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py\", line 124, in main\r\n File \"/tmp/ansible_kolla_container_facts_payload_msvskycx/ansible_kolla_container_facts_payload.zip/ansible/modules/kolla_container_facts.py\", line 105, in use_podman\r\n File \"/usr/local/lib/python3.9/site-packages/podman/domain/manager.py\", line 68, in reload\r\n latest = self.manager.get(self.id)\r\n File \"/usr/local/lib/python3.9/site-packages/podman/domain/containers_manager.py\", line 44, in get\r\n response.raise_for_status()\r\n File \"/usr/local/lib/python3.9/site-packages/podman/api/client.py\", line 75, in raise_for_status\r\n raise APIError(cause, response=self._response, explanation=message)\r\npodman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (getting container from store \"1127b70a453e15c7b5e2ba8da307f8b17791e6a23fb2b2e94a4d5dd5fce59907\": container not known)\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 } Regards, William