<div dir="ltr">Thanks<div>This "disable_compute_service_check_for_ffu" option is not available in xena, correct ?</div><div>Cheers, Massimo</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 25, 2022 at 5:15 PM Sean Mooney <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, 2022-02-25 at 16:50 +0100, Massimo Sgaravatto wrote:<br>
> I had the chance to repeat this test<br>
> So the scenario is:<br>
> <br>
> 1) controller and compute nodes running train<br>
> 2) all services stopped in compute nodes<br>
> 3) controller updated: train-->ussuri-->victoria--> wallaby<br>
> <br>
> After that nova conductor and nova scheduler refuses to start [*]<br>
<br>
yes nova does not offially support n to n+3 upgrade<br>
we started enforcing that a few release ago.<br>
there is a workaround config option that we recently added that turns<br>
the error into a waring <a href="https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu</a><br>
that is one option or you can implement or before you upgrade the contoler you can force-down all the comptue nodes<br>
<br>
<br>
<br>
> <br>
> At that moment nova-compute services were not running on the compute nodes<br>
> And this was the status on the services table:<br>
> <br>
> mysql> select * from services where topic="compute";<br>
> +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>
> > created_at | updated_at | deleted_at | id | host<br>
> | binary | topic | report_count | disabled |<br>
> deleted | disabled_reason | last_seen_up |<br>
> forced_down | version | uuid |<br>
> +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>
> > 2018-01-11 17:20:34 | 2022-02-25 09:09:17 | NULL | 17 |<br>
> <a href="http://compute-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">compute-01.cloud.pd.infn.it</a> | nova-compute | compute | 10250811 |<br>
> 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:13 |<br>
> 0 | 40 | 2f56b8cf-1190-4999-af79-6bcee695c653 |<br>
> > 2018-01-11 17:26:39 | 2022-02-25 09:09:49 | NULL | 23 |<br>
> <a href="http://compute-02.cloud.pd.infn.it" rel="noreferrer" target="_blank">compute-02.cloud.pd.infn.it</a> | nova-compute | compute | 10439622 |<br>
> 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:09:49 |<br>
> 0 | 40 | fbe37dfd-4a6c-4da1-96e0-407f7f98c4c4 |<br>
> > 2018-01-11 17:27:12 | 2022-02-25 09:10:02 | NULL | 24 |<br>
> <a href="http://compute-03.cloud.pd.infn.it" rel="noreferrer" target="_blank">compute-03.cloud.pd.infn.it</a> | nova-compute | compute | 10361295 |<br>
> 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 |<br>
> 0 | 40 | 3675f324-81dd-445a-b4eb-510726104be3 |<br>
> > 2021-04-06 12:54:42 | 2022-02-25 09:10:02 | NULL | 25 |<br>
> <a href="http://compute-04.cloud.pd.infn.it" rel="noreferrer" target="_blank">compute-04.cloud.pd.infn.it</a> | nova-compute | compute | 1790955 |<br>
> 1 | 0 | AUTO: Connection to libvirt lost: 1 | 2022-02-25 09:10:02 |<br>
> 0 | 40 | e3e7af4d-b25b-410c-983e-8128a5e97219 |<br>
> +---------------------+---------------------+------------+----+-----------------------------+--------------+---------+--------------+----------+---------+-------------------------------------+---------------------+-------------+---------+--------------------------------------+<br>
> 4 rows in set (0.00 sec)<br>
> <br>
> <br>
> <br>
> Only after manually setting the version field of these entries to '54',<br>
> nova-conductor and nova-scheduler were able to start<br>
> <br>
> Regards, Massimo<br>
> <br>
> <br>
> [*]<br>
> 2022-02-25 15:06:03.992 591600 CRITICAL nova<br>
> [req-cc20f294-cced-434b-98cd-5bdf228a2a22 - - - - -] Unhandled error:<br>
> nova.exception.TooOldComputeService: Current Nova ve<br>
> rsion does not support computes older than Wallaby but the minimum compute<br>
> service level in your system is 40 and the oldest supported service level<br>
> is 54.<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova Traceback (most recent call last):<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova File "/usr/bin/nova-conductor",<br>
> line 10, in <module><br>
> 2022-02-25 15:06:03.992 591600 ERROR nova sys.exit(main())<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova File<br>
> "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova topic=rpcapi.RPC_TOPIC)<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova File<br>
> "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in create<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova utils.raise_if_old_compute()<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova File<br>
> "/usr/lib/python3.6/site-packages/nova/utils.py", line 1098, in<br>
> raise_if_old_compute<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova<br>
> oldest_supported_service=oldest_supported_service_level)<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova<br>
> nova.exception.TooOldComputeService: Current Nova version does not support<br>
> computes older than Wallaby but the minimum comput<br>
> e service level in your system is 40 and the oldest supported service level<br>
> is 54.<br>
> 2022-02-25 15:06:03.992 591600 ERROR nova<br>
> <br>
> <br>
> <br>
> On Mon, Jan 10, 2022 at 6:00 PM Massimo Sgaravatto <<br>
> <a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> wrote:<br>
> <br>
> > Dear all<br>
> > <br>
> > When we upgraded our Cloud from Rocky to Train we followed the following<br>
> > procedure:<br>
> > <br>
> > 1) Shutdown of all services on the controller and compute nodes<br>
> > 2) Update from Rocky to Stein of controller (just to do the dbsyncs)<br>
> > 3) Update from Stein to Train of controller<br>
> > 4) Update from Rocky to Train of compute nodes<br>
> > <br>
> > We are trying to do the same to update from Train to Xena, but now there<br>
> > is a problem because<br>
> > nova services on the controller node refuse to start since they find too<br>
> > old compute nodes (this is indeed a new feature, properly documented in the<br>
> > release notes).<br>
> > As a workaround we had to manually modify the "version" field of the<br>
> > compute nodes in the nova.services table.<br>
> > <br>
> > Is it ok, or is there a cleaner way to manage the issue ?<br>
> > <br>
> > Thanks, Massimo<br>
> > <br>
<br>
</blockquote></div>