so just a suggestion i would basiclly try turning it off and on a agin. so i think the problem is that you limit the reconfig to nova but the srrro seam to indicate they placement is not runing or it not a new enough verions. so in epoxy i belive kolla has a stop command so i woudl be tempted to do a stop, pull the latest version of the kolla images with the pull action and then do anohter deploy rather then reconfigure. recornigure at least in the past is actully just ment to update the config file and restart the contiers in place. deploy does that but it can also update to newever version fo the images if you pull them first. i have not worked on kolla in a very long time nad i rearly update my home deployment so the kolla team may have a better suggestion but that would be what i tired if it was my home cluster. note that when doing deploy i woudl allow it to run for all service his is effectivly jsut doing a mindor update to latest stable release of epoxy. i would also make sure your kolla-ansible is using the latest epoxy version too incase your are hitting a fix know issue. thre is now a 20.1.0 aviable. if your using it form git you can obviously just do a git pull but done forget to make sure you updated the deps pip3 install --upgrade 'ansible-core>=2.17,<2.18.99' kolla-ansible==20.1.0 kolla-ansible install-deps i think the root of your problem is with the fact palcement is not starting properly adn returnningn 500s the python application found error i think is because of setuptools 80 and the the way the wsig script used to be generate becore we added a pyrpoject.toml so im hoping using the latest images will have that resolved. my home clustier is still on caracal and i need to upgrade it soon but i cant confirm if that will solve the issue but it likely that it will. On 18/07/2025 08:27, Franck VEDEL (UGA) wrote:
Hello everyone, I need your help. II will try to explain my problem by providing a lot of information. I have been trying to solve it for 3 days, without success. In May, my old cluster became too small for the 200 students using it, so I started installing a new cluster. Six nodes, three in one data center, three in another. After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration, and I thought everything would be ready by September.
In May, my old cluster was becoming too small for the 200 students using it, so I started installing a new cluster. 6 nodes, 3 in one data center, 3 in another. After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration; I thought everything would be ready by September. At the beginning of July, I was told to shut down everything, due to electrical maintenance in a data center. I shut everything down. 3 days later, I restarted it, and nothing worked anymore. However, it was simply a shutdown/restart of the servers.
The problems are with Nova. Here's some information and logs, if you can help me restart the cluster.
So Epoxy on Ubuntu 2404.
(venv2) *user1@ordi1*:*~/OPENSTACK/INSTALL*$ kolla-ansible --version
kolla-ansible 20.0.1.dev4
Since nova didn't want to work I did this:
(venv2) *user1@ordi1*:*~/OPENSTACK/INSTALL*$ kolla-ansible reconfigure -I multinode -t nova
ERROR with a task:
TASK [nova-cell : Waiting for nova-compute services to register themselves] ****************
skipping: [ordi2]
skipping: [ordi3]
skipping: [ordi4]
skipping: [ordi5]
skipping: [ordi6]
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
fatal: [ordi1]: FAILED! => {"msg": "Unexpected failure during module execution: Expecting value: line 1 column 1 (char 0)", "stdout": ""}
I search with Google and find this (BUG2 in the link): https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
But I am in 20.0.1.dev4
There is a lot of logs (with nova)…..
total 533340
-rw-r--r-- 1 42436 42436 0 juil. 17 06:42 nova-api.log
-rw-r--r-- 1 42436 42436 105512247 juil. 17 06:42 nova-api.log.1
-rw-r--r-- 1 42436 42436 2460643 juil. 6 06:40 *nova-api.log.2.gz*
-rw-r--r-- 1 42436 42436 7305818 juil. 5 06:31 *nova-api.log.3.gz*
-rw-r--r-- 1 42436 42436 7234505 juil. 2 06:48 *nova-api.log.4.gz*
-rw-r--r-- 1 42436 42436 2356307 juin 29 06:35 *nova-api.log.5.gz*
-rw-r--r-- 1 42436 42436 7181854 juin 28 06:49 *nova-api.log.6.gz*
-rw-r--r-- 1 42436 42436 56065234 juil. 18 09:06 nova-api-uwsgi.log
-rw-r--r-- 1 42436 42436 113689390 juil. 17 06:42 nova-api-uwsgi.log.1
-rw-r--r-- 1 42436 42436 2150056 juil. 6 06:40 *nova-api-uwsgi.log.2.gz*
-rw-r--r-- 1 42436 42436 6393602 juil. 5 06:31 *nova-api-uwsgi.log.3.gz*
-rw-r--r-- 1 42436 42436 6383183 juil. 2 06:48 *nova-api-uwsgi.log.4.gz*
-rw-r--r-- 1 42436 42436 2083376 juin 29 06:35 *nova-api-uwsgi.log.5.gz*
-rw-r--r-- 1 42436 42436 6351824 juin 28 06:49 *nova-api-uwsgi.log.6.gz*
-rw-r--r-- 1 42436 42436 1467596 juil. 18 09:06 nova-compute.log
-rw-r--r-- 1 42436 42436 2626598 juil. 18 09:06 nova-conductor.log
-rw-r--r-- 1 42436 42436 44883046 juil. 18 06:45 nova-conductor.log.1
-rw-r--r-- 1 42436 42436 7154 juil. 18 08:58 nova-manage.log
-rw-r--r-- 1 42436 42436 221050 juil. 16 15:08 nova-metadata.log
-rw-r--r-- 1 42436 42436 28094556 juil. 18 09:06 nova-metadata-uwsgi.log
-rw-r--r-- 1 42436 42436 74174166 juil. 17 06:42 nova-metadata-uwsgi.log.1
-rw-r--r-- 1 42436 42436 5650339 juil. 5 06:31 *nova-metadata-uwsgi.log.2.gz*
-rw-r--r-- 1 42436 42436 1861983 juin 29 06:35 *nova-metadata-uwsgi.log.3.gz*
-rw-r--r-- 1 42436 42436 5601477 juin 27 06:53 *nova-metadata-uwsgi.log.4.gz*
-rw-r--r-- 1 42436 42436 5640648 juin 21 06:36 *nova-metadata-uwsgi.log.5.gz*
-rw-r--r-- 1 42436 42436 1871248 juin 15 06:43 *nova-metadata-uwsgi.log.6.gz*
-rw-r--r-- 1 42436 42436 111880 juil. 16 15:08 nova-novncproxy.log
-rw-r--r-- 1 42436 42436 2668833 juil. 18 09:06 nova-scheduler.log
-rw-r--r-- 1 42436 42436 45997165 juil. 18 06:45 nova-scheduler.log.1
-rw-r--r-- 1 42436 42436 0 mai 15 08:47 privsep-helper.log
2 containers have problems:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6231ee0f3c97 quay.io/openstack.kolla/nova-scheduler:2025.1-ubuntu-noble "dumb-init --single-…" 10 minutes ago Exited (1) 5 seconds ago nova_scheduler
57707a9d136e quay.io/openstack.kolla/nova-conductor:2025.1-ubuntu-noble "dumb-init --single-…" 39 hours ago Exited (1) 4 seconds ago nova_conductor
Have a look to nova_conductor.log (same thing with nova_scheduler):
2025-07-18 09:08:25.170 7 ERROR nova.scheduler.client.report [-] Failed to initialize placement client (is keystone available?): openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.
2025-07-18 09:08:25.171 7 ERROR nova.conductor.manager [-] Fatal error initializing placement client: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.
2025-07-18 09:08:25.172 7 CRITICAL nova [-] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.
2025-07-18 09:08:25.172 7 ERROR nova Traceback (most recent call last):
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/bin/nova-conductor", line 8, in <module>
2025-07-18 09:08:25.172 7 ERROR nova sys.exit(main())
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/cmd/conductor.py", line 45, in main
2025-07-18 09:08:25.172 7 ERROR nova server = service.Service.create(binary='nova-conductor',
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 256, in create
2025-07-18 09:08:25.172 7 ERROR nova service_obj = cls(host, binary, topic, manager,
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 120, in __init__
2025-07-18 09:08:25.172 7 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 124, in __init__
2025-07-18 09:08:25.172 7 ERROR nova self.compute_task_mgr = ComputeTaskManager()
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 255, in __init__
2025-07-18 09:08:25.172 7 ERROR nova self.report_client
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 283, in report_client
2025-07-18 09:08:25.172 7 ERROR nova return report.report_client_singleton()
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 91, in report_client_singleton
2025-07-18 09:08:25.172 7 ERROR nova PLACEMENTCLIENT = SchedulerReportClient()
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 237, in __init__
2025-07-18 09:08:25.172 7 ERROR nova self._client = self._create_client()
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 280, in _create_client
2025-07-18 09:08:25.172 7 ERROR nova client = self._adapter or utils.get_sdk_adapter(
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/utils.py", line 1035, in get_sdk_adapter
2025-07-18 09:08:25.172 7 ERROR nova return getattr(conn, service_type.replace('-', '_'))
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 88, in __get__
2025-07-18 09:08:25.172 7 ERROR nova proxy = self._make_proxy(instance)
2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 290, in _make_proxy
2025-07-18 09:08:25.172 7 ERROR nova raise exceptions.NotSupported(
2025-07-18 09:08:25.172 7 ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.
2025-07-18 09:08:25.172 7 ERROR nova
Another thing is strange:
Autre point bizarre:
tail -f /var/log/kolla/placement/placement-api-uwsgi.log
--- no python application found, check your startup logs for errors ---
[pid: 1092|app: -1|req: -1/307186] 192.168.100.4 () {22 vars in 215 bytes} [Fri Jul 18 09:11:28 2025] GET / => generated 21 bytes in 0 msecs (HTTP/1.0 500) 3 headers in 102 bytes (0 switches on core 0)
What can I do ? Can you help me to find a solution to restart my Openstack ? Thanks a lot if you can.
Franck