Hi *,
last week I upgraded my lab environment from Caracal to Epoxy. It's a
package based deployment, so no kolla or ansible involved, but we use
pacemaker for HA. The distro was upgraded from 22.04 to 24.04 as well.
The upgrade of Openstack itself went quite smoothly, no issues at all.
But for some reason, the neutron-metadata-agents don't update their
status in the neutron db:
MariaDB [(none)]> select
id,`binary`,admin_state_up,heartbeat_timestamp,resources_synced from
neutron.agents;
+--------------------------------------+---------------------------+----------------+---------------------+------------------+
| id | binary |
admin_state_up | heartbeat_timestamp | resources_synced |
+--------------------------------------+---------------------------+----------------+---------------------+------------------+
| 0a5f439e-a31c-4e93-8861-6c4f97d354c0 | neutron-openvswitch-agent |
1 | 2025-07-23 07:29:43 | NULL |
| 0e3cea86-2cc3-455f-8eb4-e14a9897cad3 | neutron-dhcp-agent |
1 | 2025-07-23 07:29:59 | NULL |
| 19c944fb-bdc1-4aa0-9612-7794f924da3d | neutron-openvswitch-agent |
1 | 2025-07-23 07:30:00 | NULL |
| 20709aa0-2c55-4a18-8f7f-2c65c1bc1297 | neutron-dhcp-agent |
1 | 2025-07-23 07:29:48 | NULL |
| 4d6c8bff-eb30-46ca-becd-9a8cbb08e9cc | neutron-l3-agent |
1 | 2025-07-23 07:30:00 | NULL |
| a52fb631-8fc3-4707-9fca-06165c56b481 | neutron-metadata-agent |
1 | 2025-07-17 09:37:14 | NULL |
| aa3bd2e4-e3dd-4ad5-bc69-963e68e8567f | neutron-l3-agent |
1 | 2025-07-23 07:29:53 | NULL |
| e8f8f679-f881-445b-b207-4beee679f6d3 | neutron-openvswitch-agent |
1 | 2025-07-23 07:29:53 | NULL |
| f04f347d-f376-4afc-99bc-0b4cd1c4c76a | neutron-metadata-agent |
1 | 2025-07-17 09:50:40 | NULL |
+--------------------------------------+---------------------------+----------------+---------------------+------------------+
So they're also shown as not alive:
# openstack network agent list
Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has
no attribute 'OldListMessages'
Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has
no attribute 'OldPostMessages'
+--------------------------------------+--------------------+--------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host
| Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+--------------+-------------------+-------+-------+---------------------------+
| 0a5f439e-a31c-4e93-8861-6c4f97d354c0 | Open vSwitch agent |
controller02 | None | :-) | UP |
neutron-openvswitch-agent |
| 0e3cea86-2cc3-455f-8eb4-e14a9897cad3 | DHCP agent |
controller01 | nova | :-) | UP | neutron-dhcp-agent
|
| 19c944fb-bdc1-4aa0-9612-7794f924da3d | Open vSwitch agent |
controller01 | None | :-) | UP |
neutron-openvswitch-agent |
| 20709aa0-2c55-4a18-8f7f-2c65c1bc1297 | DHCP agent |
controller02 | nova | :-) | UP | neutron-dhcp-agent
|
| 4d6c8bff-eb30-46ca-becd-9a8cbb08e9cc | L3 agent |
controller02 | nova | :-) | UP | neutron-l3-agent
|
| a52fb631-8fc3-4707-9fca-06165c56b481 | Metadata agent |
controller01 | None | XXX | UP |
neutron-metadata-agent |
| aa3bd2e4-e3dd-4ad5-bc69-963e68e8567f | L3 agent |
controller01 | nova | :-) | UP | neutron-l3-agent
|
| e8f8f679-f881-445b-b207-4beee679f6d3 | Open vSwitch agent |
compute002 | None | :-) | UP |
neutron-openvswitch-agent |
| f04f347d-f376-4afc-99bc-0b4cd1c4c76a | Metadata agent |
controller02 | None | XXX | UP |
neutron-metadata-agent |
+--------------------------------------+--------------------+--------------+-------------------+-------+-------+---------------------------+
I turned on debug logs, nothing pops up in the logs, and the daemons
work. Launching a new VM produces expected output in the agent log
file, but I'll spare you the debug logs. The instance can reach the
metadata service:
$ curl http://169.254.169.254/latest/meta-data/instance-id
i-0000000f
RabbitMQ accepts the connections as well:
2025-07-18 19:33:27.773935+00:00 [info] <0.7940.1> connection
<0.7940.1> (192.168.160.8:60856 -> 192.168.160.9:5672) has a
client-provided name:
neutron-metadata-agent:987277:5352e341-42df-4a92-a03f-1a58afab53a1
2025-07-18 19:33:27.775762+00:00 [info] <0.7940.1> connection
<0.7940.1> (192.168.160.8:60856 -> 192.168.160.9:5672 -
neutron-metadata-agent:987277:5352e341-42df-4a92-a03f-1a58afab53a1):
user 'openstack' authenticated and granted access to vhost 'openstack'
So I'm wondering what I might be missing here. I'd appreciate any pointers!
Thanks!
Eugen