[kolla-ansible][nova] nova upgrade failed from zed to antelope
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result] ************************************************************************************************************************************************************ fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
Le mer. 21 juin 2023 à 06:16, Satish Patel satish.txt@gmail.com a écrit :
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
As you can read in the documentation below, yes indeed it became mandatory by the 27.1.0 release.
In general, if you have a question about a new release, you can look at the release notes by https://docs.openstack.org/releasenotes/nova/2023.1.html#relnotes-27-1-0-sta... and you'll see what is changed.
HTH, -Sylvain
[1]
https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
Hi Satish,
It very strange because the fix for service token for the Nova was merged a month ago ( https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f) Maybe you have custom configuration which overrides nova.conf ?
On Wed, Jun 21, 2023 at 7:16 AM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
Hi Maksim,
This is all I have in my config/ I don't have any override.
(venv-kolla) root@kolla-infra-1:~# ls -l /etc/kolla/config/ total 8 -rw-r--r-- 1 root root 187 Apr 30 04:11 global.conf drwxr-xr-x 2 root root 4096 May 3 01:38 neutron
(venv-kolla) root@kolla-infra-1:~# cat /etc/kolla/config/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 rabbit_transient_queues_ttl=60
On Wed, Jun 21, 2023 at 4:15 AM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
It very strange because the fix for service token for the Nova was merged a month ago ( https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f) Maybe you have custom configuration which overrides nova.conf ?
On Wed, Jun 21, 2023 at 7:16 AM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
-- Regards, Maksim Malchuk
Seems strange. If you have the correct latest images, and can reproduce the issue, then create the bug report on launchpad.
On Wed, Jun 21, 2023 at 4:02 PM Satish Patel satish.txt@gmail.com wrote:
Hi Maksim,
This is all I have in my config/ I don't have any override.
(venv-kolla) root@kolla-infra-1:~# ls -l /etc/kolla/config/ total 8 -rw-r--r-- 1 root root 187 Apr 30 04:11 global.conf drwxr-xr-x 2 root root 4096 May 3 01:38 neutron
(venv-kolla) root@kolla-infra-1:~# cat /etc/kolla/config/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 rabbit_transient_queues_ttl=60
On Wed, Jun 21, 2023 at 4:15 AM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
It very strange because the fix for service token for the Nova was merged a month ago ( https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f) Maybe you have custom configuration which overrides nova.conf ?
On Wed, Jun 21, 2023 at 7:16 AM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
-- Regards, Maksim Malchuk
Just to close this loop, after adding the following in /etc/kolla/nova-api/nova.conf fixed my issue. But still need to understand what went wrong in upgrade path
[service_user] send_service_user_token = true auth_url = http://10.30.50.10:5000 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = nova password = AAABBBCCCDDDEEE cafile = region_name = RegionOne valid_interfaces = internal
On Wed, Jun 21, 2023 at 9:02 AM Satish Patel satish.txt@gmail.com wrote:
Hi Maksim,
This is all I have in my config/ I don't have any override.
(venv-kolla) root@kolla-infra-1:~# ls -l /etc/kolla/config/ total 8 -rw-r--r-- 1 root root 187 Apr 30 04:11 global.conf drwxr-xr-x 2 root root 4096 May 3 01:38 neutron
(venv-kolla) root@kolla-infra-1:~# cat /etc/kolla/config/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 rabbit_transient_queues_ttl=60
On Wed, Jun 21, 2023 at 4:15 AM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
It very strange because the fix for service token for the Nova was merged a month ago ( https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f) Maybe you have custom configuration which overrides nova.conf ?
On Wed, Jun 21, 2023 at 7:16 AM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
-- Regards, Maksim Malchuk
Hi,
they just added check for service user , check the log :
+---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
If we want to fix this in kolla-ansible, we should generate another config file for container which is actually running nova-upgrade status check, run container with this config,
and if it will pass it's ok.
Problem is that we are running upgrade check before config generation.
If you run kolla-ansible genconfig before upgrade on host1 it will pass also...
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 21. 6. 2023 v 16:54 odesílatel Satish Patel satish.txt@gmail.com napsal:
Just to close this loop, after adding the following in /etc/kolla/nova-api/nova.conf fixed my issue. But still need to understand what went wrong in upgrade path
[service_user] send_service_user_token = true auth_url = http://10.30.50.10:5000 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = nova password = AAABBBCCCDDDEEE cafile = region_name = RegionOne valid_interfaces = internal
On Wed, Jun 21, 2023 at 9:02 AM Satish Patel satish.txt@gmail.com wrote:
Hi Maksim,
This is all I have in my config/ I don't have any override.
(venv-kolla) root@kolla-infra-1:~# ls -l /etc/kolla/config/ total 8 -rw-r--r-- 1 root root 187 Apr 30 04:11 global.conf drwxr-xr-x 2 root root 4096 May 3 01:38 neutron
(venv-kolla) root@kolla-infra-1:~# cat /etc/kolla/config/global.conf [oslo_messaging_rabbit] kombu_reconnect_delay=0.5 rabbit_transient_queues_ttl=60
On Wed, Jun 21, 2023 at 4:15 AM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
It very strange because the fix for service token for the Nova was merged a month ago ( https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f) Maybe you have custom configuration which overrides nova.conf ?
On Wed, Jun 21, 2023 at 7:16 AM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'am upgrading zed to antelope using kolla-ansible and encount following error
TASK [nova : Upgrade status check result]
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]} fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-chec... "]}
After running upgrade check command manually on nova-api container got following error
(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. +---------------------------------------------------------------------+ | Upgrade Check Results | +---------------------------------------------------------------------+ | Check: Cells v2 | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Placement API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Cinder API | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Policy File JSON to YAML Migration | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Older than N-1 computes | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: hw_machine_type unset | | Result: Success | | Details: None | +---------------------------------------------------------------------+ | Check: Service User Token Configuration | | Result: Failure | | Details: Service user token configuration is required for all Nova | | services. For more details see the following: https://docs | | .openstack.org/latest/nova/admin/configuration/service- | | user-token.html | +---------------------------------------------------------------------+
Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova?
[1] https://docs.openstack.org/nova/latest/admin/configuration/service-user-toke...
-- Regards, Maksim Malchuk
participants (4)
-
Maksim Malchuk
-
Michal Arbet
-
Satish Patel
-
Sylvain Bauza