Hi! I'm facing a weird situation where Ceilometer compute agent fails to get libvirt domain metadata on an Ubuntu 20.04 LTS with the latest updates, kernel 5.4.0-65-generic and Openstack Wallaby Nova compute services installed using the official Wallaby repo for Ubuntu 20.04. All components have been deployed manually. Ceilometer agent is configured with instance_discovery_method = libvirt_metadata. The agent is unable to fetch the domain metadata, and the following error messages appear in /var/log/ceilometer/ceilometer-agent-compute.log on agent start and periodic polling attempts: 2021-06-01 16:01:18.297 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid baf06f57-ac5b-4661-928c-7adaeaea0311 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.298 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 208c0d7a-41a3-4fa6-bf72-2f9594ac6b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.300 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid d979a527-c1ba-4b29-8e30-322d4d2efcf7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.301 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid a41f21b6-766d-4979-bbe1-84f421b0c3f2 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid fd5ffe32-c6d6-4898-9ba2-2af1ffebd502 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid aff042c9-c311-4944-bc42-09ccd5a90eb7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.303 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 9510bc46-e4e2-490c-9cbe-c9eb5e349b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.304 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 4d2c2c9b-4eff-460a-a00b-19fdbe33f5d4 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present 2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu_l3_cache, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster memory.usage, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177 All domains exist and their metadata is readily available using virsh or a simple Python script. Nova compute service is fully functional, Ceilometer agent is partially functional as it is able to export compute.node.cpu.* metrics but nothing related to libvirt domains. I already filed a bug report https://bugs.launchpad.net/ceilometer/+bug/1930446, but would appreciate feedback and/or advice. Best regards, Zakhar --- More deployment information: Installed Ceilometer-related packages: ceilometer-agent-compute 2:16.0.0-0ubuntu1~cloud0 ceilometer-common 2:16.0.0-0ubuntu1~cloud0 python3-ceilometer 2:16.0.0-0ubuntu1~cloud0 Installed Nova-related packages: nova-common 3:23.0.0-0ubuntu1~cloud0 nova-compute 3:23.0.0-0ubuntu1~cloud0 nova-compute-kvm 3:23.0.0-0ubuntu1~cloud0 nova-compute-libvirt 3:23.0.0-0ubuntu1~cloud0 python3-nova 3:23.0.0-0ubuntu1~cloud0 python3-novaclient 2:17.4.0-0ubuntu1~cloud0 Installed Libvirt-related packages: libvirt-clients 6.0.0-0ubuntu8.9 libvirt-daemon 6.0.0-0ubuntu8.9 libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9 libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9 libvirt-daemon-system 6.0.0-0ubuntu8.9 libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9 libvirt0:amd64 6.0.0-0ubuntu8.9 python3-libvirt 6.1.0-1 Installed Qemu-related packages: libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9 qemu-block-extra:amd64 1:4.2-3ubuntu6.16 qemu-kvm 1:4.2-3ubuntu6.16 qemu-system-common 1:4.2-3ubuntu6.16 qemu-system-data 1:4.2-3ubuntu6.16 qemu-system-gui:amd64 1:4.2-3ubuntu6.16 qemu-system-x86 1:4.2-3ubuntu6.16 qemu-utils 1:4.2-3ubuntu6.16 Apparmor is enabled and running the default configuration, no messages related to apparmor and libvirt, qemu, nova-compute, ceilometer-agent, etc are visible in the logs. I am also attaching the relevant Ceilometer agent and Nova configuration files: ceilometer.conf: [DEFAULT] transport_url = rabbit://WORKING-TRANSPORT-URL verbose = true debug = true auth_strategy = keystone log_dir = /var/log/ceilometer [compute] instance_discovery_method = libvirt_metadata [keystone_authtoken] www_authenticate_uri = http://CONTROLLER.VIP:5000 auth_url = http://CONTROLLER.VIP:5000 memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS auth_type = password project_domain_name = default user_domain_name = default project_name = service username = ceilometer password = WORKING_PASSWORD [service_credentials] auth_type = password auth_url = http://CONTROLLER.VIP:5000/v3 memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS project_domain_id = default user_domain_id = default project_name = service username = ceilometer password = WORKING_PASSWORD interface = internalURL region_name = RegionOne [oslo_messaging_notifications] driver = messagingv2 transport_url = rabbit://WORKING-TRANSPORT-URL polling.yaml: --- sources: - name: some_pollsters interval: 300 meters: - cpu - cpu_l3_cache - memory.usage - network.incoming.bytes - network.incoming.packets - network.outgoing.bytes - network.outgoing.packets - disk.device.read.bytes - disk.device.read.requests - disk.device.write.bytes - disk.device.write.requests - hardware.cpu.util - hardware.cpu.user - hardware.cpu.nice - hardware.cpu.system - hardware.cpu.idle - hardware.cpu.wait - hardware.cpu.kernel - hardware.cpu.interrupt - hardware.memory.used - hardware.memory.total - hardware.memory.buffer - hardware.memory.cached - hardware.memory.swap.avail - hardware.memory.swap.total - hardware.system_stats.io.outgoing.blocks - hardware.system_stats.io.incoming.blocks - hardware.network.ip.incoming.datagrams - hardware.network.ip.outgoing.datagrams nova.conf: [DEFAULT] log_dir = /var/log/nova lock_path = /var/lock/nova state_path = /var/lib/nova instance_usage_audit_period = hour compute_monitors = cpu.virt_driver,numa_mem_bw.virt_driver reserved_host_memory_mb = 2048 instance_usage_audit = True resume_guests_state_on_host_boot = true my_ip = COMPUTE.HOST.IP.ADDR report_interval = 30 transport_url = rabbit://WORKING-TRANSPORT-URL [api] [api_database] [barbican] [cache] expiration_time = 600 backend = oslo_cache.memcache_pool backend_argument = memcached_expire_time:660 enabled = true memcache_servers = LIST-OF-WORKING-MEMCACHED-SERVERS [cinder] catalog_info = volumev3::internalURL [compute] [conductor] [console] [consoleauth] [cors] [cyborg] [database] connection = mysql+pymysql://WORKING-CONNECTION-STRING connection_recycle_time = 280 max_pool_size = 5 max_retries = -1 [devices] [ephemeral_storage_encryption] [filter_scheduler] [glance] api_servers = http://CONTROLLER.VIP:9292 [guestfs] [healthcheck] [hyperv] [image_cache] [ironic] [key_manager] [keystone] [keystone_authtoken] www_authenticate_uri = http://CONTROLLER.VIP:5000 auth_url = http://CONTROLLER.VIP:5000 region_name = RegionOne memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = nova password = WORKING-PASSWORD [libvirt] live_migration_scheme = ssh live_migration_permit_post_copy = true disk_cachemodes="network=writeback,block=writeback" images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = cinder rbd_secret_uuid = SECRET-UUID [metrics] [mks] [neutron] auth_url = http://CONTROLLER.VIP:5000 region_name = RegionOne memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = neutron password = WORKING-PASSWORD [notifications] notify_on_state_change = vm_and_task_state [oslo_concurrency] lock_path = /var/lib/nova/tmp [oslo_messaging_amqp] [oslo_messaging_kafka] [oslo_messaging_notifications] driver = messagingv2 [oslo_messaging_rabbit] amqp_auto_delete = false rabbit_ha_queues = true [oslo_middleware] [oslo_policy] [pci] [placement] region_name = RegionOne project_domain_name = Default project_name = service auth_type = password user_domain_name = Default auth_url = http://CONTROLLER.VIP:5000/v3 username = placement password = WORKING-PASSWORD [powervm] [privsep] [profiler] [quota] [rdp] [remote_debug] [scheduler] [serial_console] [service_user] [spice] [upgrade_levels] [vault] [vendordata_dynamic_auth] [vmware] [vnc] enabled = true server_listen = 0.0.0.0 server_proxyclient_address = $my_ip novncproxy_base_url = https://WORKING-URL:6080/vnc_auto.html [workarounds] [wsgi] [zvm] [cells] enable = False [os_region_name] openstack =