On Fri, 2022-02-25 at 16:50 +0100, Massimo Sgaravatto wrote:
I had the chance to repeat this test So the scenario is:
1) controller and compute nodes running train 2) all services stopped in compute nodes 3) controller updated: train-->ussuri-->victoria--> wallaby
After that nova conductor and nova scheduler refuses to start [*]
yes nova does not offially support n to n+3 upgrade we started enforcing that a few release ago. there is a workaround config option that we recently added that turns the error into a waring https://docs.openstack.org/nova/latest/configuration/config.html#workarounds... that is one option or you can implement or before you upgrade the contoler you can force-down all the comptue nodes
At that moment nova-compute services were not running on the compute nodes And this was the status on the services table:
mysql> select * from services where topic="compute"; +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+
created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid | +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+ 2018-01-11 17:20:34 | 2022-02-25 09:09:17 | NULL | 17 | compute-01.cloud.pd.infn.it | nova-compute | compute | 10250811 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:13 | 0 | 40 | 2f56b8cf-1190-4999-af79-6bcee695c653 | 2018-01-11 17:26:39 | 2022-02-25 09:09:49 | NULL | 23 | compute-02.cloud.pd.infn.it | nova-compute | compute | 10439622 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:49 | 0 | 40 | fbe37dfd-4a6c-4da1-96e0-407f7f98c4c4 | 2018-01-11 17:27:12 | 2022-02-25 09:10:02 | NULL | 24 | compute-03.cloud.pd.infn.it | nova-compute | compute | 10361295 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | 3675f324-81dd-445a-b4eb-510726104be3 | 2021-04-06 12:54:42 | 2022-02-25 09:10:02 | NULL | 25 | compute-04.cloud.pd.infn.it | nova-compute | compute | 1790955 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | e3e7af4d-b25b-410c-983e-8128a5e97219 | +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+ 4 rows in set (0.00 sec)
Only after manually setting the version field of these entries to '54', nova-conductor and nova-scheduler were able to start
Regards, Massimo
[*] 2022-02-25 15:06:03.992 591600 CRITICAL nova [req-cc20f294-cced-434b-98cd-5bdf228a2a22 - - - - -] Unhandled error: nova.exception.TooOldComputeService: Current Nova ve rsion does not support computes older than Wallaby but the minimum compute service level in your system is 40 and the oldest supported service level is 54. 2022-02-25 15:06:03.992 591600 ERROR nova Traceback (most recent call last): 2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module> 2022-02-25 15:06:03.992 591600 ERROR nova sys.exit(main()) 2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main 2022-02-25 15:06:03.992 591600 ERROR nova topic=rpcapi.RPC_TOPIC) 2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in create 2022-02-25 15:06:03.992 591600 ERROR nova utils.raise_if_old_compute() 2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/utils.py", line 1098, in raise_if_old_compute 2022-02-25 15:06:03.992 591600 ERROR nova oldest_supported_service=oldest_supported_service_level) 2022-02-25 15:06:03.992 591600 ERROR nova nova.exception.TooOldComputeService: Current Nova version does not support computes older than Wallaby but the minimum comput e service level in your system is 40 and the oldest supported service level is 54. 2022-02-25 15:06:03.992 591600 ERROR nova
On Mon, Jan 10, 2022 at 6:00 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Dear all
When we upgraded our Cloud from Rocky to Train we followed the following procedure:
1) Shutdown of all services on the controller and compute nodes 2) Update from Rocky to Stein of controller (just to do the dbsyncs) 3) Update from Stein to Train of controller 4) Update from Rocky to Train of compute nodes
We are trying to do the same to update from Train to Xena, but now there is a problem because nova services on the controller node refuse to start since they find too old compute nodes (this is indeed a new feature, properly documented in the release notes). As a workaround we had to manually modify the "version" field of the compute nodes in the nova.services table.
Is it ok, or is there a cleaner way to manage the issue ?
Thanks, Massimo