PFA On Wed, Jul 30, 2025 at 3:30 PM TRB PREM KUMAR <kumar6009@gmail.com> wrote:

Hi Community,

I added 2 more container logs (/var/etc/kolla/ceilometer) for reference. Please check and let me know

Thanks, TRB.

On Wed, Jul 30, 2025 at 2:55 PM TRB PREM KUMAR <kumar6009@gmail.com> wrote:

...

Hi, We have deployed an all-in-one OpenStack server for billing. Now ceilometer containers are unhealthy. Please help us to resolve this issue. Reference Output for reference: trb1@openstack-aio-01:~$ docker ps | grep ceil c53fc69ba42b quay.io/openstack.kolla/ceilometer-compute:2024.2-rocky-9 "dumb-init --single-…" About a minute ago Up About a minute *(unhealthy) * ceilometer_compute 9337af646e0f quay.io/openstack.kolla/ceilometer-central:2024.2-rocky-9 "dumb-init --single-…" 30 minutes ago Up About a minute ( *unhealthy)* ceilometer_central 554c4b8ccd86 quay.io/openstack.kolla/ceilometer-notification:2024.2-rocky-9 "dumb-init --single-…" 4 days ago Up About a minute (healthy) ceilometer_notification

*** Logs files are attached for reference.

Regards, TRB.

It looks like the same problem is on Launchpad: https://bugs.launchpad.net/kolla-ansible/+bug/2081190 On Wed, Jul 30, 2025 at 2:09 PM Vivekanandan Muthukrishnan < vivek@vnetpops.com> wrote:

Hi TRB,

This issue appears to be related to the health check bash script used within the Kolla Ansible container.

In the image quay.io/openstack.kolla/ceilometer-compute:2024.2-rocky-9, the ss command does not provide process details with PIDs as expected. As a result, the final command in the script */usr/local/bin/healthcheck_port* fails, which in turn causes the Docker container's health check to report an unhealthy state.

To resolve this, you can simply comment out or modify the final line in the health check script to bypass the PID check, which is currently not functioning as intended in this image.

# Just hack to make the container healthy, i guess this has nothing to do with the ceilometer operations. cat /usr/local/bin/healthcheck_port #!/bin/bash process=$1

shift 1 args=$@ ports=${args// /|} pids=$(pgrep -d '|' -f $process) #ss -ntp | grep -qE ":($ports).*,pid=($pids),"

Otherwise, you can even disable the docker container health check as part of tagging the docker image.

Regards Vivek

On Wed, Jul 30, 2025 at 2:57 PM TRB PREM KUMAR <kumar6009@gmail.com> wrote:

...

Hi, We have deployed an all-in-one OpenStack server for billing. Now ceilometer containers are unhealthy. Please help us to resolve this issue. Reference Output for reference: trb1@openstack-aio-01:~$ docker ps | grep ceil c53fc69ba42b quay.io/openstack.kolla/ceilometer-compute:2024.2-rocky-9 "dumb-init --single-…" About a minute ago Up About a minute *(unhealthy) * ceilometer_compute 9337af646e0f quay.io/openstack.kolla/ceilometer-central:2024.2-rocky-9 "dumb-init --single-…" 30 minutes ago Up About a minute ( *unhealthy)* ceilometer_central 554c4b8ccd86 quay.io/openstack.kolla/ceilometer-notification:2024.2-rocky-9 "dumb-init --single-…" 4 days ago Up About a minute (healthy) ceilometer_notification

*** Logs files are attached for reference.

Regards, TRB.

-- Regards, Maksim Malchuk

Dear Team, *Subject: *Horizon UI Crash Due to Custom compute Service in CloudKitty Hashmap We’re encountering an issue with the Horizon dashboard (/admin/hashmap/) crashing after creating a custom service in CloudKitty’s hashmap. *Issue Summary:* - Environment: OpenStack with Kolla-Ansible - Service Added: compute (via openstack rating hashmap service create compute) - Result: Horizon's Hashmap Admin UI crashes with a 500 error. *Analysis:* The Horizon container logs show the following traceback: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittydashboard/api/client.py", line 211, in __init__ data = list(self.client.metric.get()) ... keystoneauth1.exceptions.http.HttpError: {"faultcode": "Client", "faultstring": "compute", "debuginfo": null} (HTTP 404) *This indicates that Horizon attempts to call the /v1/info/metric/compute endpoint in CloudKitty, which returns 404 because compute is not a valid metric name. *Relevant Log Snippet (/var/log/kolla/horizon/horizon-error.log):* keystoneauth1.exceptions.http.HttpError: {"faultcode": "Client", "faultstring": "compute", "debuginfo": null} (HTTP 404) Full log attached below : horizon-error.log *Temporary Fix Applied:* Deleted the invalid service with: openstack rating hashmap service delete 9873a779-c023-424c-b8c1-5150b663da5b docker restart horizon Horizon /admin/hashmap/ is now loading without errors. *Request:* Please confirm whether the name compute is reserved and must match the actual metric names reported by Gnocchi/Ceilometer. Suggest any validation mechanism to prevent the creation of invalid services in HashMap that crash Horizon. Thank you for your support. Regards, TRB. *Full Log for Reference: * root@openstack-aio-01:/home/trb1# cat /var/log/kolla/horizon/horizon-error.log | tail -n 100 2025-08-01 07:53:41.177674 self._data = {self.table_class._meta.name: self.get_data()} 2025-08-01 07:53:41.177675 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittydashboard/dashboards/admin/hashmap/views.py", line 42, in get_data 2025-08-01 07:53:41.177677 service = manager.info.get_metric(metric_name=s['name']) 2025-08-01 07:53:41.177678 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/utils.py", line 71, in wrap 2025-08-01 07:53:41.177679 raise http.HttpError(message=e.response.text, 2025-08-01 07:53:41.177680 keystoneauth1.exceptions.http.HttpError: {"faultcode": "Client", "faultstring": "compute", "debuginfo": null} (HTTP 404) 2025-08-01 07:53:47.519763 /var/lib/kolla/venv/lib/python3.9/site-packages/oslo_policy/policy.py:1134: UserWarning: Policy "os_compute_api:servers:create": "rule:project_member_or_admin" failed scope check. The token used to make the request was domain scoped but the policy requires ['project'] scope. This behavior may change in the future where using the intended scope is required 2025-08-01 07:53:47.519782 warnings.warn(msg) 2025-08-01 07:53:50.590161 Internal Server Error: /admin/hashmap/ 2025-08-01 07:53:50.590179 Traceback (most recent call last): 2025-08-01 07:53:50.590181 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/utils.py", line 69, in wrap 2025-08-01 07:53:50.590191 return func(*args, **kwargs) 2025-08-01 07:53:50.590193 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/v1/info.py", line 32, in get_metric 2025-08-01 07:53:50.590194 return self.api_client.get(url).json() 2025-08-01 07:53:50.590196 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 393, in get 2025-08-01 07:53:50.590197 return self.request(url, 'GET', **kwargs) 2025-08-01 07:53:50.590198 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 255, in request 2025-08-01 07:53:50.590200 return self.session.request(url, method, **kwargs) 2025-08-01 07:53:50.590201 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/session.py", line 985, in request 2025-08-01 07:53:50.590203 raise exceptions.from_response(resp, method, url) 2025-08-01 07:53:50.590204 keystoneauth1.exceptions.http.NotFound: Unrecognized schema in response body. (HTTP 404) (Request-ID: req-dd9f0bfd-792b-4e09-8019-0ca89182b92d) 2025-08-01 07:53:50.590206 2025-08-01 07:53:50.590207 During handling of the above exception, another exception occurred: 2025-08-01 07:53:50.590209 2025-08-01 07:53:50.590210 Traceback (most recent call last): 2025-08-01 07:53:50.590212 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner 2025-08-01 07:53:50.590214 response = get_response(request) 2025-08-01 07:53:50.590215 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response 2025-08-01 07:53:50.590217 response = wrapped_callback(request, *callback_args, **callback_kwargs) 2025-08-01 07:53:50.590219 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 51, in dec 2025-08-01 07:53:50.590220 return view_func(request, *args, **kwargs) 2025-08-01 07:53:50.590222 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 35, in dec 2025-08-01 07:53:50.590223 return view_func(request, *args, **kwargs) 2025-08-01 07:53:50.590225 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 35, in dec 2025-08-01 07:53:50.590227 return view_func(request, *args, **kwargs) 2025-08-01 07:53:50.590228 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 111, in dec 2025-08-01 07:53:50.590230 return view_func(request, *args, **kwargs) 2025-08-01 07:53:50.590232 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 104, in view 2025-08-01 07:53:50.590234 return self.dispatch(request, *args, **kwargs) 2025-08-01 07:53:50.590235 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch 2025-08-01 07:53:50.590236 return handler(request, *args, **kwargs) 2025-08-01 07:53:50.590239 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 222, in get 2025-08-01 07:53:50.590241 handled = self.construct_tables() 2025-08-01 07:53:50.590243 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 213, in construct_tables 2025-08-01 07:53:50.590245 handled = self.handle_table(table) 2025-08-01 07:53:50.590246 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 122, in handle_table 2025-08-01 07:53:50.590248 data = self._get_data_dict() 2025-08-01 07:53:50.590250 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 251, in _get_data_dict 2025-08-01 07:53:50.590252 self._data = {self.table_class._meta.name: self.get_data()} 2025-08-01 07:53:50.590253 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittydashboard/dashboards/admin/hashmap/views.py", line 42, in get_data 2025-08-01 07:53:50.590255 service = manager.info.get_metric(metric_name=s['name']) 2025-08-01 07:53:50.590256 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/utils.py", line 71, in wrap 2025-08-01 07:53:50.590262 raise http.HttpError(message=e.response.text, 2025-08-01 07:53:50.590280 keystoneauth1.exceptions.http.HttpError: {"faultcode": "Client", "faultstring": "compute", "debuginfo": null} (HTTP 404) 2025-08-01 07:56:43.521254 Internal Server Error: /admin/hashmap/ 2025-08-01 07:56:43.521273 Traceback (most recent call last): 2025-08-01 07:56:43.521276 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/utils.py", line 69, in wrap 2025-08-01 07:56:43.521278 return func(*args, **kwargs) 2025-08-01 07:56:43.521279 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/v1/info.py", line 32, in get_metric 2025-08-01 07:56:43.521281 return self.api_client.get(url).json() 2025-08-01 07:56:43.521283 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 393, in get 2025-08-01 07:56:43.521284 return self.request(url, 'GET', **kwargs) 2025-08-01 07:56:43.521286 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 255, in request 2025-08-01 07:56:43.521288 return self.session.request(url, method, **kwargs) 2025-08-01 07:56:43.521290 File "/var/lib/kolla/venv/lib/python3.9/site-packages/keystoneauth1/session.py", line 985, in request 2025-08-01 07:56:43.521291 raise exceptions.from_response(resp, method, url) 2025-08-01 07:56:43.521293 keystoneauth1.exceptions.http.NotFound: Unrecognized schema in response body. (HTTP 404) (Request-ID: req-c7667804-df1a-4e7c-aa51-cb3df9186972) 2025-08-01 07:56:43.521295 2025-08-01 07:56:43.521296 During handling of the above exception, another exception occurred: 2025-08-01 07:56:43.521298 2025-08-01 07:56:43.521299 Traceback (most recent call last): 2025-08-01 07:56:43.521301 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner 2025-08-01 07:56:43.521303 response = get_response(request) 2025-08-01 07:56:43.521304 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response 2025-08-01 07:56:43.521306 response = wrapped_callback(request, *callback_args, **callback_kwargs) 2025-08-01 07:56:43.521308 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 51, in dec 2025-08-01 07:56:43.521310 return view_func(request, *args, **kwargs) 2025-08-01 07:56:43.521311 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 35, in dec 2025-08-01 07:56:43.521313 return view_func(request, *args, **kwargs) 2025-08-01 07:56:43.521315 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 35, in dec 2025-08-01 07:56:43.521316 return view_func(request, *args, **kwargs) 2025-08-01 07:56:43.521318 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/decorators.py", line 111, in dec 2025-08-01 07:56:43.521319 return view_func(request, *args, **kwargs) 2025-08-01 07:56:43.521321 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 104, in view 2025-08-01 07:56:43.521323 return self.dispatch(request, *args, **kwargs) 2025-08-01 07:56:43.521324 File "/var/lib/kolla/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch 2025-08-01 07:56:43.521326 return handler(request, *args, **kwargs) 2025-08-01 07:56:43.521327 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 222, in get 2025-08-01 07:56:43.521329 handled = self.construct_tables() 2025-08-01 07:56:43.521331 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 213, in construct_tables 2025-08-01 07:56:43.521332 handled = self.handle_table(table) 2025-08-01 07:56:43.521334 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 122, in handle_table 2025-08-01 07:56:43.521335 data = self._get_data_dict() 2025-08-01 07:56:43.521337 File "/var/lib/kolla/venv/lib/python3.9/site-packages/horizon/tables/views.py", line 251, in _get_data_dict 2025-08-01 07:56:43.521347 self._data = {self.table_class._meta.name: self.get_data()} 2025-08-01 07:56:43.521348 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittydashboard/dashboards/admin/hashmap/views.py", line 42, in get_data 2025-08-01 07:56:43.521350 service = manager.info.get_metric(metric_name=s['name']) 2025-08-01 07:56:43.521351 File "/var/lib/kolla/venv/lib/python3.9/site-packages/cloudkittyclient/utils.py", line 71, in wrap 2025-08-01 07:56:43.521352 raise http.HttpError(message=e.response.text, 2025-08-01 07:56:43.521353 keystoneauth1.exceptions.http.HttpError: {"faultcode": "Client", "faultstring": "compute", "debuginfo": null} (HTTP 404)