<div dir="ltr"><div dir="ltr">I had the chance to repeat this test<br></div><div>So the scenario is:</div><div><br></div><div>1) controller and compute nodes running train</div><div>2) all services stopped in compute nodes</div><div>3) controller updated: train-->ussuri-->victoria--> wallaby</div><div><br></div><div>After that nova conductor and nova scheduler refuses to start [*]</div><div><br></div><div>At that moment nova-compute services were not running on the compute nodes</div><div>And this was the status on the services table:</div><div><br></div><div>mysql> select * from services where topic="compute";<br>+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |<br>+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>| 2018-01-11 17:20:34 | 2022-02-25 09:09:17 | NULL | 17 | <a href="http://compute-01.cloud.pd.infn.it">compute-01.cloud.pd.infn.it</a> | nova-compute | compute | 10250811 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:13 | 0 | 40 | 2f56b8cf-1190-4999-af79-6bcee695c653 |<br>| 2018-01-11 17:26:39 | 2022-02-25 09:09:49 | NULL | 23 | <a href="http://compute-02.cloud.pd.infn.it">compute-02.cloud.pd.infn.it</a> | nova-compute | compute | 10439622 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:49 | 0 | 40 | fbe37dfd-4a6c-4da1-96e0-407f7f98c4c4 |<br>| 2018-01-11 17:27:12 | 2022-02-25 09:10:02 | NULL | 24 | <a href="http://compute-03.cloud.pd.infn.it">compute-03.cloud.pd.infn.it</a> | nova-compute | compute | 10361295 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | 3675f324-81dd-445a-b4eb-510726104be3 |<br>| 2021-04-06 12:54:42 | 2022-02-25 09:10:02 | NULL | 25 | <a href="http://compute-04.cloud.pd.infn.it">compute-04.cloud.pd.infn.it</a> | nova-compute | compute | 1790955 | 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 | 0 | 40 | e3e7af4d-b25b-410c-983e-8128a5e97219 |<br>+---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>4 rows in set (0.00 sec)<br><br><br><br>Only after manually setting the version field of these entries to '54', nova-conductor and nova-scheduler were able to start <br><br></div><div>Regards, Massimo</div><div><br></div><div><br></div><div>[*]</div><div>2022-02-25 15:06:03.992 591600 CRITICAL nova
[req-cc20f294-cced-434b-98cd-5bdf228a2a22 - - - - -] Unhandled error:
nova.exception.TooOldComputeService: Current Nova ve<br>
rsion does not support computes older than Wallaby but the minimum
compute service level in your system is 40 and the oldest supported
service level is 54.<br>
2022-02-25 15:06:03.992 591600 ERROR nova Traceback (most recent call last):<br>
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module><br>
2022-02-25 15:06:03.992 591600 ERROR nova sys.exit(main())<br>
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main<br>
2022-02-25 15:06:03.992 591600 ERROR nova topic=rpcapi.RPC_TOPIC)<br>
2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in create<br>
2022-02-25 15:06:03.992 591600 ERROR nova utils.raise_if_old_compute()<br>
2022-02-25 15:06:03.992 591600 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/utils.py", line 1098, in
raise_if_old_compute<br>
2022-02-25 15:06:03.992 591600 ERROR nova oldest_supported_service=oldest_supported_service_level)<br>
2022-02-25 15:06:03.992 591600 ERROR nova
nova.exception.TooOldComputeService: Current Nova version does not
support computes older than Wallaby but the minimum comput<br>
e service level in your system is 40 and the oldest supported service level is 54.<br>
2022-02-25 15:06:03.992 591600 ERROR nova <br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 10, 2022 at 6:00 PM Massimo Sgaravatto <<a href="mailto:massimo.sgaravatto@gmail.com">massimo.sgaravatto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear all<br><div><br></div><div>When we upgraded our Cloud from Rocky to Train we followed the following procedure:</div><div><br></div><div>1) Shutdown of all services on the controller and compute nodes</div><div>2) Update from Rocky to Stein of controller (just to do the dbsyncs)</div><div>3) Update from Stein to Train of controller</div><div>4) Update from Rocky to Train of compute nodes</div><div><br></div><div>We are trying to do the same to update from Train to Xena, but now there is a problem because</div><div> nova services on the controller node refuse to start since they find too old compute nodes (this is indeed a new feature, properly documented in the release notes).</div><div>As a workaround we had to manually modify the "version" field of the compute nodes in the nova.services table.</div><div><br></div><div>Is it ok, or is there a cleaner way to manage the issue ?</div><div><br></div><div>Thanks, Massimo</div></div>
</blockquote></div></div>