Just to close this loop, after adding the following in /etc/kolla/nova-api/nova.conf fixed my issue. But still need to understand what went wrong in upgrade path

[service_user]
send_service_user_token = true
auth_url = http://10.30.50.10:5000
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = AAABBBCCCDDDEEE
cafile =
region_name = RegionOne
valid_interfaces = internal

On Wed, Jun 21, 2023 at 9:02 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Maksim,

This is all I have in my config/  I don't have any override.

(venv-kolla) root@kolla-infra-1:~# ls -l /etc/kolla/config/
total 8
-rw-r--r-- 1 root root  187 Apr 30 04:11 global.conf
drwxr-xr-x 2 root root 4096 May  3 01:38 neutron


(venv-kolla) root@kolla-infra-1:~# cat /etc/kolla/config/global.conf
[oslo_messaging_rabbit]
kombu_reconnect_delay=0.5
rabbit_transient_queues_ttl=60



On Wed, Jun 21, 2023 at 4:15 AM Maksim Malchuk <maksim.malchuk@gmail.com> wrote:
Hi Satish,

It very strange because the fix for service token for the Nova was merged a month ago (https://review.opendev.org/q/I2189dafca070accfd8efcd4b8cc4221c6decdc9f)
Maybe you have custom configuration which overrides nova.conf ?

On Wed, Jun 21, 2023 at 7:16 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,

I'am upgrading zed to antelope using kolla-ansible and encount following error 

TASK [nova : Upgrade status check result] ************************************************************************************************************************************************************
fatal: [kolla-infra-1]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-checks"]}
fatal: [kolla-infra-2]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-checks"]}
fatal: [kolla-infra-3]: FAILED! => {"changed": false, "msg": ["There was an upgrade status check failure!", "See the detail at https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-checks"]}


After running upgrade check command manually on nova-api container got following error 

(nova-api)[root@kolla-infra-2 /]# nova-status upgrade check
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
+---------------------------------------------------------------------+
| Upgrade Check Results                                               |
+---------------------------------------------------------------------+
| Check: Cells v2                                                     |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: Placement API                                                |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: Cinder API                                                   |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: Policy File JSON to YAML Migration                           |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: Older than N-1 computes                                      |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: hw_machine_type unset                                        |
| Result: Success                                                     |
| Details: None                                                       |
+---------------------------------------------------------------------+
| Check: Service User Token Configuration                             |
| Result: Failure                                                     |
| Details:  Service user token configuration is required for all Nova |
|   services. For more details see the following: https://docs        |
|   .openstack.org/latest/nova/admin/configuration/service-           |
|   user-token.html                                                   |
+---------------------------------------------------------------------+

Service user token reference to the following doc [1] . Do I need to configure token users in order to upgrade nova? 





--
Regards,
Maksim Malchuk