Hello, What could I be doing wrong here? When we attempt openstack deploy, this step fails with the error message below? kolla-ansible -i /etc/kolla/multinode deploy debug3: mux_client_request_alive: done pid = 91641 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 The full traceback is: File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py", line 425, in main File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 515, in recreate_or_restart_container self.start_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 538, in start_container self.create_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 501, in create_container container = self.pc.containers.create(**args) File "/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py", line 354, in create response.raise_for_status(not_found=ImageNotFound) File "/usr/local/lib/python3.9/site-packages/podman/api/client.py", line 75, in raise_for_status raise APIError(cause, response=self._response, explanation=message) fatal: [ostack-40.openstack.perasotech.com]: FAILED! => { "changed": true, "invocation": { "module_args": { "action": "recreate_or_restart_container", "api_version": "auto", "auth_email": null, "auth_password": null, "auth_registry": "quay.io", "auth_username": null, "client_timeout": 120, "container_engine": "podman", "detach": true, "dimensions": { "ulimits": {} }, "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS" }, "graceful_timeout": 60, "ignore_missing": false, "image": "quay.io/openstack.kolla/fluentd:master-rocky-9", "labels": {}, "name": "fluentd", "privileged": false, "remove_on_exit": true, "restart_policy": "unless-stopped", "restart_retries": 10, "security_opt": [], "state": "running", "tls_verify": false, "tty": false } }, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py\", line 425, in main\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 515, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 538, in start_container\\n self.create_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 501, in create_container\\n container = self.pc.containers.create(**args)\\n File \"/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py\", line 354, in create\\n response.raise_for_status(not_found=ImageNotFound)\\n File \"/usr/local/lib/python3.9/site-packages/podman/api/client.py\", line 75, in raise_for_status\\n raise APIError(cause, response=self._response, explanation=message)\\npodman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (statfs /var/log/journal: no such file or directory)\\n'" } Have anyone see the issue before? Regards, William
Hello William! I looked at the error you provided. According to the traceback at the end of the output, it looks like this is the problem that podman reported: /var/log/journal: no such file or directory. Can you check to see if the /var/log/journal directory exists or if podman has the permissions to access the systemd journal directory? Can you start podman containers outside of Kolla Ansible without any errors? I hope that helps. Good luck with this issue! Thank you, Jadon On Wed, Jul 17, 2024 at 8:09 AM William Muriithi <wmuriithi@perasoinc.com> wrote:
Hello,
What could I be doing wrong here?
When we attempt openstack deploy, this step fails with the error message below?
kolla-ansible -i /etc/kolla/multinode deploy
debug3: mux_client_request_alive: done pid = 91641 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 The full traceback is: File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py", line 425, in main File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 515, in recreate_or_restart_container self.start_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 538, in start_container self.create_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 501, in create_container container = self.pc.containers.create(**args) File "/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py", line 354, in create response.raise_for_status(not_found=ImageNotFound) File "/usr/local/lib/python3.9/site-packages/podman/api/client.py", line 75, in raise_for_status raise APIError(cause, response=self._response, explanation=message) fatal: [ostack-40.openstack.perasotech.com]: FAILED! => { "changed": true, "invocation": { "module_args": { "action": "recreate_or_restart_container", "api_version": "auto", "auth_email": null, "auth_password": null, "auth_registry": "quay.io", "auth_username": null, "client_timeout": 120, "container_engine": "podman", "detach": true, "dimensions": { "ulimits": {} }, "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS" }, "graceful_timeout": 60, "ignore_missing": false, "image": "quay.io/openstack.kolla/fluentd:master-rocky-9", "labels": {}, "name": "fluentd", "privileged": false, "remove_on_exit": true, "restart_policy": "unless-stopped", "restart_retries": 10, "security_opt": [], "state": "running", "tls_verify": false, "tty": false } }, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py\", line 425, in main\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 515, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 538, in start_container\\n self.create_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 501, in create_container\\n container = self.pc.containers.create(**args)\\n File \"/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py\", line 354, in create\\n response.raise_for_status(not_found=ImageNotFound)\\n File \"/usr/local/lib/python3.9/site-packages/podman/api/client.py\", line 75, in raise_for_status\\n raise APIError(cause, response=self._response, explanation=message)\\npodman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (statfs /var/log/journal: no such file or directory)\\n'" }
Have anyone see the issue before?
Regards, William
Jadon, Thanks a lot. The problem was there was no /var/log/journal directory. Not sure how that happened on both systems as that directory should have been created by systemd. But creating it manually allowed me to move forward. Thanks again a lot. William From: Jadon Naas <jadon.naas@canonical.com> Sent: Wednesday, July 17, 2024 10:27 AM To: William Muriithi <wmuriithi@perasoinc.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: Kolla deploy You don't often get email from jadon.naas@canonical.com<mailto:jadon.naas@canonical.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hello William! I looked at the error you provided. According to the traceback at the end of the output, it looks like this is the problem that podman reported: /var/log/journal: no such file or directory. Can you check to see if the /var/log/journal directory exists or if podman has the permissions to access the systemd journal directory? Can you start podman containers outside of Kolla Ansible without any errors? I hope that helps. Good luck with this issue! Thank you, Jadon On Wed, Jul 17, 2024 at 8:09 AM William Muriithi <wmuriithi@perasoinc.com<mailto:wmuriithi@perasoinc.com>> wrote: Hello, What could I be doing wrong here? When we attempt openstack deploy, this step fails with the error message below? kolla-ansible -i /etc/kolla/multinode deploy debug3: mux_client_request_alive: done pid = 91641 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 The full traceback is: File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py", line 425, in main File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 515, in recreate_or_restart_container self.start_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 538, in start_container self.create_container() File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 501, in create_container container = self.pc.containers.create(**args) File "/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py", line 354, in create response.raise_for_status(not_found=ImageNotFound) File "/usr/local/lib/python3.9/site-packages/podman/api/client.py", line 75, in raise_for_status raise APIError(cause, response=self._response, explanation=message) fatal: [ostack-40.openstack.perasotech.com<http://ostack-40.openstack.perasotech.com>]: FAILED! => { "changed": true, "invocation": { "module_args": { "action": "recreate_or_restart_container", "api_version": "auto", "auth_email": null, "auth_password": null, "auth_registry": "quay.io<http://quay.io>", "auth_username": null, "client_timeout": 120, "container_engine": "podman", "detach": true, "dimensions": { "ulimits": {} }, "environment": { "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS" }, "graceful_timeout": 60, "ignore_missing": false, "image": "quay.io/openstack.kolla/fluentd:master-rocky-9<http://quay.io/openstack.kolla/fluentd:master-rocky-9>", "labels": {}, "name": "fluentd", "privileged": false, "remove_on_exit": true, "restart_policy": "unless-stopped", "restart_retries": 10, "security_opt": [], "state": "running", "tls_verify": false, "tty": false } }, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py\", line 425, in main\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 515, in recreate_or_restart_container\\n self.start_container()\\n<file://n> File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 538, in start_container\\n self.create_container()\\n<file://n> File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 501, in create_container\\n container = self.pc.containers.create(**args)\\n<file://n> File \"/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py\", line 354, in create\\n response.raise_for_status(not_found=ImageNotFound)\\n<file://n> File \"/usr/local/lib/python3.9/site-packages/podman/api/client.py\", line 75, in raise_for_status\\n raise APIError(cause, response=self._response, explanation=message)\\npodman.errors.exceptions.APIError<file://npodman.errors.exceptions.APIError>: 500 Server Error: Internal Server Error (statfs /var/log/journal: no such file or directory)\\n<file://n>'" } Have anyone see the issue before? Regards, William
Hello William! I'm glad to hear you got it resolved and that the problem was the missing /var/log/journal directory. It is strange that the directory did not exist. Thanks for writing back to share that you fixed it and how you fixed it! Thank you! Jadon On Thu, Jul 18, 2024 at 12:28 PM William Muriithi <wmuriithi@perasoinc.com> wrote:
Jadon,
Thanks a lot. The problem was there was no /var/log/journal directory.
Not sure how that happened on both systems as that directory should have been created by systemd.
But creating it manually allowed me to move forward.
Thanks again a lot.
William
*From:* Jadon Naas <jadon.naas@canonical.com> *Sent:* Wednesday, July 17, 2024 10:27 AM *To:* William Muriithi <wmuriithi@perasoinc.com> *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: Kolla deploy
You don't often get email from jadon.naas@canonical.com. Learn why this is important <https://aka.ms/LearnAboutSenderIdentification>
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hello William!
I looked at the error you provided. According to the traceback at the end of the output, it looks like this is the problem that podman reported: /var/log/journal: no such file or directory. Can you check to see if the /var/log/journal directory exists or if podman has the permissions to access the systemd journal directory? Can you start podman containers outside of Kolla Ansible without any errors?
I hope that helps. Good luck with this issue!
Thank you,
Jadon
On Wed, Jul 17, 2024 at 8:09 AM William Muriithi <wmuriithi@perasoinc.com> wrote:
Hello,
What could I be doing wrong here?
When we attempt openstack deploy, this step fails with the error message below?
kolla-ansible -i /etc/kolla/multinode deploy
debug3: mux_client_request_alive: done pid = 91641
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
The full traceback is:
File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py", line 425, in main
File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 515, in recreate_or_restart_container
self.start_container()
File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 538, in start_container
self.create_container()
File "/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py", line 501, in create_container
container = self.pc.containers.create(**args)
File "/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py", line 354, in create
response.raise_for_status(not_found=ImageNotFound)
File "/usr/local/lib/python3.9/site-packages/podman/api/client.py", line 75, in raise_for_status
raise APIError(cause, response=self._response, explanation=message)
fatal: [ostack-40.openstack.perasotech.com]: FAILED! => {
"changed": true,
"invocation": {
"module_args": {
"action": "recreate_or_restart_container",
"api_version": "auto",
"auth_email": null,
"auth_password": null,
"auth_registry": "quay.io",
"auth_username": null,
"client_timeout": 120,
"container_engine": "podman",
"detach": true,
"dimensions": {
"ulimits": {}
},
"environment": {
"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"
},
"graceful_timeout": 60,
"ignore_missing": false,
"image": "quay.io/openstack.kolla/fluentd:master-rocky-9",
"labels": {},
"name": "fluentd",
"privileged": false,
"remove_on_exit": true,
"restart_policy": "unless-stopped",
"restart_retries": 10,
"security_opt": [],
"state": "running",
"tls_verify": false,
"tty": false
}
},
"msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/modules/kolla_container.py\", line 425, in main\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 515, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 538, in start_container\\n self.create_container()\\n File \"/tmp/ansible_kolla_container_payload_xai740dx/ansible_kolla_container_payload.zip/ansible/module_utils/kolla_podman_worker.py\", line 501, in create_container\\n container = self.pc.containers.create(**args)\\n File \"/usr/local/lib/python3.9/site-packages/podman/domain/containers_create.py\", line 354, in create\\n response.raise_for_status(not_found=ImageNotFound) \\n File \"/usr/local/lib/python3.9/site-packages/podman/api/client.py\", line 75, in raise_for_status\\n raise APIError(cause, response=self._response, explanation=message)\\npodman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (statfs /var/log/journal: no such file or directory)\\n'"
}
Have anyone see the issue before?
Regards,
William
participants (2)
-
Jadon Naas
-
William Muriithi