II will try to explain my problem by providing a lot of information. I have been trying to solve it for 3 days, without success.

In May, my old cluster became too small for the 200 students using it, so I started installing a new cluster. Six nodes, three in one data center, three in another.

After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration, and I thought everything would be ready by September.

In May, my old cluster was becoming too small for the 200 students using it, so I started installing a new cluster. 6 nodes, 3 in one data center, 3 in another.

After testing with Bobcat and Caracal (unsatisfactory, with numerous issues, especially with authentication and display in Horizon), I tried Epoxy. I installed everything I needed and tested it. It was a real success. It was very fast in its new configuration; I thought everything would be ready by September.

At the beginning of July, I was told to shut down everything, due to electrical maintenance in a data center. I shut everything down. 3 days later, I restarted it, and nothing worked anymore. However, it was simply a shutdown/restart of the servers.

The problems are with Nova.

Here's some information and logs, if you can help me restart the cluster.

So Epoxy on Ubuntu 2404.

(venv2) user1@ordi1:~/OPENSTACK/INSTALL$ kolla-ansible --version

kolla-ansible 20.0.1.dev4

Since nova didn't want to work I did this:

(venv2) user1@ordi1:~/OPENSTACK/INSTALL$ kolla-ansible reconfigure -I multinode -t nova

ERROR with a task:

TASK [nova-cell : Waiting for nova-compute services to register themselves] ****************

skipping: [ordi2]

skipping: [ordi3]

skipping: [ordi4]

skipping: [ordi5]

skipping: [ordi6]

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

fatal: [ordi1]: FAILED! => {"msg": "Unexpected failure during module execution: Expecting value: line 1 column 1 (char 0)", "stdout": ""}

I search with Google and find this (BUG2 in the link):

But I am in 20.0.1.dev4

There is a lot of logs (with nova)…..

total 533340

-rw-r--r-- 1 42436 42436 0 juil. 17 06:42 nova-api.log

-rw-r--r-- 1 42436 42436 105512247 juil. 17 06:42 nova-api.log.1

-rw-r--r-- 1 42436 42436 2460643 juil. 6 06:40 nova-api.log.2.gz

-rw-r--r-- 1 42436 42436 7305818 juil. 5 06:31 nova-api.log.3.gz

-rw-r--r-- 1 42436 42436 7234505 juil. 2 06:48 nova-api.log.4.gz

-rw-r--r-- 1 42436 42436 2356307 juin 29 06:35 nova-api.log.5.gz

-rw-r--r-- 1 42436 42436 7181854 juin 28 06:49 nova-api.log.6.gz

-rw-r--r-- 1 42436 42436 56065234 juil. 18 09:06 nova-api-uwsgi.log

-rw-r--r-- 1 42436 42436 113689390 juil. 17 06:42 nova-api-uwsgi.log.1

-rw-r--r-- 1 42436 42436 2150056 juil. 6 06:40 nova-api-uwsgi.log.2.gz

-rw-r--r-- 1 42436 42436 6393602 juil. 5 06:31 nova-api-uwsgi.log.3.gz

-rw-r--r-- 1 42436 42436 6383183 juil. 2 06:48 nova-api-uwsgi.log.4.gz

-rw-r--r-- 1 42436 42436 2083376 juin 29 06:35 nova-api-uwsgi.log.5.gz

-rw-r--r-- 1 42436 42436 6351824 juin 28 06:49 nova-api-uwsgi.log.6.gz

-rw-r--r-- 1 42436 42436 1467596 juil. 18 09:06 nova-compute.log

-rw-r--r-- 1 42436 42436 2626598 juil. 18 09:06 nova-conductor.log

-rw-r--r-- 1 42436 42436 44883046 juil. 18 06:45 nova-conductor.log.1

-rw-r--r-- 1 42436 42436 7154 juil. 18 08:58 nova-manage.log

-rw-r--r-- 1 42436 42436 221050 juil. 16 15:08 nova-metadata.log

-rw-r--r-- 1 42436 42436 28094556 juil. 18 09:06 nova-metadata-uwsgi.log

-rw-r--r-- 1 42436 42436 74174166 juil. 17 06:42 nova-metadata-uwsgi.log.1

-rw-r--r-- 1 42436 42436 5650339 juil. 5 06:31 nova-metadata-uwsgi.log.2.gz

-rw-r--r-- 1 42436 42436 1861983 juin 29 06:35 nova-metadata-uwsgi.log.3.gz

-rw-r--r-- 1 42436 42436 5601477 juin 27 06:53 nova-metadata-uwsgi.log.4.gz

-rw-r--r-- 1 42436 42436 5640648 juin 21 06:36 nova-metadata-uwsgi.log.5.gz

-rw-r--r-- 1 42436 42436 1871248 juin 15 06:43 nova-metadata-uwsgi.log.6.gz

-rw-r--r-- 1 42436 42436 111880 juil. 16 15:08 nova-novncproxy.log

-rw-r--r-- 1 42436 42436 2668833 juil. 18 09:06 nova-scheduler.log

-rw-r--r-- 1 42436 42436 45997165 juil. 18 06:45 nova-scheduler.log.1

-rw-r--r-- 1 42436 42436 0 mai 15 08:47 privsep-helper.log

2 containers have problems:

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

6231ee0f3c97 quay.io/openstack.kolla/nova-scheduler:2025.1-ubuntu-noble "dumb-init --single-…" 10 minutes ago Exited (1) 5 seconds ago nova_scheduler

57707a9d136e quay.io/openstack.kolla/nova-conductor:2025.1-ubuntu-noble "dumb-init --single-…" 39 hours ago Exited (1) 4 seconds ago nova_conductor

Have a look to nova_conductor.log (same thing with nova_scheduler):

2025-07-18 09:08:25.170 7 ERROR nova.scheduler.client.report [-] Failed to initialize placement client (is keystone available?): openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.

2025-07-18 09:08:25.171 7 ERROR nova.conductor.manager [-] Fatal error initializing placement client: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.

2025-07-18 09:08:25.172 7 CRITICAL nova [-] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.

2025-07-18 09:08:25.172 7 ERROR nova Traceback (most recent call last):

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/bin/nova-conductor", line 8, in <module>

2025-07-18 09:08:25.172 7 ERROR nova sys.exit(main())

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/cmd/conductor.py", line 45, in main

2025-07-18 09:08:25.172 7 ERROR nova server = service.Service.create(binary='nova-conductor',

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 256, in create

2025-07-18 09:08:25.172 7 ERROR nova service_obj = cls(host, binary, topic, manager,

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/service.py", line 120, in __init__

2025-07-18 09:08:25.172 7 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 124, in __init__

2025-07-18 09:08:25.172 7 ERROR nova self.compute_task_mgr = ComputeTaskManager()

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 255, in __init__

2025-07-18 09:08:25.172 7 ERROR nova self.report_client

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/conductor/manager.py", line 283, in report_client

2025-07-18 09:08:25.172 7 ERROR nova return report.report_client_singleton()

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 91, in report_client_singleton

2025-07-18 09:08:25.172 7 ERROR nova PLACEMENTCLIENT = SchedulerReportClient()

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 237, in __init__

2025-07-18 09:08:25.172 7 ERROR nova self._client = self._create_client()

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/scheduler/client/report.py", line 280, in _create_client

2025-07-18 09:08:25.172 7 ERROR nova client = self._adapter or utils.get_sdk_adapter(

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/utils.py", line 1035, in get_sdk_adapter

2025-07-18 09:08:25.172 7 ERROR nova return getattr(conn, service_type.replace('-', '_'))

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 88, in __get__

2025-07-18 09:08:25.172 7 ERROR nova proxy = self._make_proxy(instance)

2025-07-18 09:08:25.172 7 ERROR nova ^^^^^^^^^^^^^^^^^^^^^^^^^^

2025-07-18 09:08:25.172 7 ERROR nova File "/var/lib/kolla/venv/lib/python3.12/site-packages/openstack/service_description.py", line 290, in _make_proxy

2025-07-18 09:08:25.172 7 ERROR nova raise exceptions.NotSupported(

2025-07-18 09:08:25.172 7 ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.100.100:RegionOne exists but does not have any supported versions.

2025-07-18 09:08:25.172 7 ERROR nova

Another thing is strange:

Autre point bizarre:

tail -f /var/log/kolla/placement/placement-api-uwsgi.log

--- no python application found, check your startup logs for errors ---

[pid: 1092|app: -1|req: -1/307186] 192.168.100.4 () {22 vars in 215 bytes} [Fri Jul 18 09:11:28 2025] GET / => generated 21 bytes in 0 msecs (HTTP/1.0 500) 3 headers in 102 bytes (0 switches on core 0)

What can I do ? Can you help me to find a solution to restart my Openstack ? Thanks a lot if you can.