I had the chance to repeat this test
So the scenario is:
1) controller and compute nodes running train
2) all services stopped in compute nodes
3) controller updated: train-->ussuri-->victoria--> wallaby
After that nova conductor and nova scheduler refuses to start [*]
At that moment nova-compute services were not running on the compute nodes
And this was the status on the services table:
mysql> select * from services where topic="compute";
+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+
| 2018-01-11 17:20:34 | 2022-02-25 09:09:17 | NULL | 17 |
compute-01.cloud.pd.infn.it | nova-compute | compute | 10250811 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:13 | 0 | 40 | 2f56b8cf-1190-4999-af79-6bcee695c653 |
| 2018-01-11 17:26:39 | 2022-02-25 09:09:49 | NULL | 23 |
compute-02.cloud.pd.infn.it | nova-compute | compute | 10439622 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:49 | 0 | 40 | fbe37dfd-4a6c-4da1-96e0-407f7f98c4c4 |
| 2018-01-11 17:27:12 | 2022-02-25 09:10:02 | NULL | 24 |
compute-03.cloud.pd.infn.it | nova-compute | compute | 10361295 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | 3675f324-81dd-445a-b4eb-510726104be3 |
| 2021-04-06 12:54:42 | 2022-02-25 09:10:02 | NULL | 25 |
compute-04.cloud.pd.infn.it | nova-compute | compute | 1790955 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | e3e7af4d-b25b-410c-983e-8128a5e97219 |
+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+
4 rows in set (0.00 sec)
Only after manually setting the version field of these entries to '54', nova-conductor and nova-scheduler were able to start
Regards, Massimo
[*]
2022-02-25 15:06:03.992 591600 CRITICAL nova
[req-cc20f294-cced-434b-98cd-5bdf228a2a22 - - - - -] Unhandled error:
nova.exception.TooOldComputeService: Current Nova ve
rsion does not support computes older than Wallaby but the minimum
compute service level in your system is 40 and the oldest supported
service level is 54.
2022-02-25 15:06:03.992 591600 ERROR nova Traceback (most recent call last):
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2022-02-25 15:06:03.992 591600 ERROR nova sys.exit(main())
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main
2022-02-25 15:06:03.992 591600 ERROR nova topic=rpcapi.RPC_TOPIC)
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in create
2022-02-25 15:06:03.992 591600 ERROR nova utils.raise_if_old_compute()
2022-02-25 15:06:03.992 591600 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/utils.py", line 1098, in
raise_if_old_compute
2022-02-25 15:06:03.992 591600 ERROR nova oldest_supported_service=oldest_supported_service_level)
2022-02-25 15:06:03.992 591600 ERROR nova
nova.exception.TooOldComputeService: Current Nova version does not
support computes older than Wallaby but the minimum comput
e service level in your system is 40 and the oldest supported service level is 54.
2022-02-25 15:06:03.992 591600 ERROR nova
Dear all
When we upgraded our Cloud from Rocky to Train we followed the following procedure:
1) Shutdown of all services on the controller and compute nodes
2) Update from Rocky to Stein of controller (just to do the dbsyncs)
3) Update from Stein to Train of controller
4) Update from Rocky to Train of compute nodes
We are trying to do the same to update from Train to Xena, but now there is a problem because
nova services on the controller node refuse to start since they find too old compute nodes (this is indeed a new feature, properly documented in the release notes).
As a workaround we had to manually modify the "version" field of the compute nodes in the nova.services table.
Is it ok, or is there a cleaner way to manage the issue ?
Thanks, Massimo