Hi,
I am using nova 23.0.0 and I am randomly facing unauthorized error sometime when nova uses neutron or cinder phython3 client during listing of server and creation of server. When I list server again, it works fine. I have found below logs in nova-api.
2021-12-20 09:10:24.920 2689471 DEBUG nova.api.openstack.wsgi [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x7f840b6c5070>>' _process_stack /usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py:513 2021-12-20 09:10:24.925 2689471 DEBUG nova.compute.api [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Searching by: {'project_id': '890eb2b7d1b8488aa88de7c34d08817a', 'deleted': False} get_all /usr/lib/python3/dist-packages/nova/compute/api.py:2857 2021-12-20 09:10:24.930 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:355 2021-12-20 09:10:24.931 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default def2021-12-20 09:10:24.548 2689471 INFO nova.osapi_compute.wsgi.server [req-c5da1a3c-3c99-46c2-84e9-b504ccc5d156 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] 172.16.30.56 "GET /v2.1/ HTTP/1.1" status: 200 len: 800 time: 0.0048089 2021-12-20 09:10:24.697 2689471 DEBUG nova.osapi_compute.wsgi.server [req-572eca81-8d3b-4524-89e5-e4e1124a7ac5 - - - - -] (2689471) accepted ('172.16.30.56', 52016) server /usr/lib/pytho n3/dist-packages/eventlet/wsgi.py:992 2021-12-20 09:10:24.920 2689471 DEBUG nova.api.openstack.wsgi [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default defaul t] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x7f840b6c5070>>' _process_stack /usr/lib/python3/dist-packag es/nova/api/openstack/wsgi.py:513 2021-12-20 09:10:24.925 2689471 DEBUG nova.compute.api [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Sear ching by: {'project_id': '890eb2b7d1b8488aa88de7c34d08817a', 'deleted': False} get_all /usr/lib/python3/dist-packages/nova/compute/api.py:2857 2021-12-20 09:10:24.930 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default def ault] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3/d ist-packages/oslo_concurrency/lockutils.py:355 2021-12-20 09:10:24.931 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default def ault] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:367 2021-12-20 09:10:24.933 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Lock "fc805771-7891-4095-8162-491cf824b02f" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:355 2021-12-20 09:10:24.934 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Lock "fc805771-7891-4095-8162-491cf824b02f" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:367 2021-12-20 09:10:25.019 2689471 DEBUG nova.compute.multi_cell_list [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Listed batch of 13 results from cell out of 21 limit. Returned 13 total so far. do_query /usr/lib/python3/dist-packages/nova/compute/multi_cell_list.py:371 2021-12-20 09:10:25.093 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Lock "fc805771-7891-4095-8162-491cf824b02f" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:355 2021-12-20 09:10:25.094 2689471 DEBUG oslo_concurrency.lockutils [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Lock "fc805771-7891-4095-8162-491cf824b02f" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:367 *2021-12-20 09:10:25.393 2689471 DEBUG neutronclient.v2_0.client [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Error message: {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}} _handle_fault_response /usr/lib/python3/dist-packages/neutronclient/v2_0/client.py:258* 2021-12-20 09:10:25.394 2689471 ERROR nova.network.neutron [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: neutronclient.common.exceptions.Unauthorized: 401-{'error': {'code': 401, 'title': 'Unauthorized', 'message': 'The request you have made requires authentication.'}} 2021-12-20 09:10:25.395 2689471 INFO nova.api.openstack.wsgi [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] HTTP exception thrown: Networking client is experiencing an unauthorized exception. 2021-12-20 09:10:25.396 2689471 DEBUG nova.api.openstack.wsgi [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] Returning 400 to user: Networking client is experiencing an unauthorized exception. __call__ /usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py:927 2021-12-20 09:10:25.398 2689471 INFO nova.osapi_compute.wsgi.server [req-91ee9c88-e8ab-4e9b-8e1a-ca8a807c7e45 2af528fdf3244e15b4f3f8fcfc0889c5 890eb2b7d1b8488aa88de7c34d08817a - default default] 172.16.30.56 "GET /v2.1/890eb2b7d1b8488aa88de7c34d08817a/servers/detail?limit=21&project_id=890eb2b7d1b8488aa88de7c34d08817a&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_key=display_name&sort_key=uuid HTTP/1.1" status: 400 len: 529 time: 0.7002470
Is there anything that I need to change ?