[telemetry] Wallaby ceilometer.compute.discovery fails to get domain metadata

Zakhar Kirpichenko zakhar at gmail.com
Wed Jun 2 03:24:55 UTC 2021


Hi!

I'm facing a weird situation where Ceilometer compute agent fails to get
libvirt domain metadata on an Ubuntu 20.04 LTS with the latest updates,
kernel 5.4.0-65-generic and Openstack Wallaby Nova compute services
installed using the official Wallaby repo for Ubuntu 20.04. All components
have been deployed manually.

Ceilometer agent is configured with instance_discovery_method =
libvirt_metadata. The agent is unable to fetch the domain metadata, and the
following error messages appear in
/var/log/ceilometer/ceilometer-agent-compute.log on agent start and
periodic polling attempts:

2021-06-01 16:01:18.297 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid baf06f57-ac5b-4661-928c-7adaeaea0311 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.298 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid 208c0d7a-41a3-4fa6-bf72-2f9594ac6b8d metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.300 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid d979a527-c1ba-4b29-8e30-322d4d2efcf7 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.301 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid a41f21b6-766d-4979-bbe1-84f421b0c3f2 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid fd5ffe32-c6d6-4898-9ba2-2af1ffebd502 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid aff042c9-c311-4944-bc42-09ccd5a90eb7 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.303 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid 9510bc46-e4e2-490c-9cbe-c9eb5e349b8d metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.304 1835684 ERROR ceilometer.compute.discovery [-] Fail
to get domain uuid 4d2c2c9b-4eff-460a-a00b-19fdbe33f5d4 metadata,
libvirtError: metadata not found: Requested metadata element is not
present: libvirt.libvirtError: metadata not found: Requested metadata
element is not present
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster cpu_l3_cache, no resources found this cycle poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster disk.device.write.bytes, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster network.incoming.packets, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster disk.device.read.requests, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster network.outgoing.packets, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster network.outgoing.bytes, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster network.incoming.bytes, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster cpu, no resources found this cycle poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster disk.device.write.requests, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster disk.device.read.bytes, no resources found this cycle
poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip
pollster memory.usage, no resources found this cycle poll_and_notify
/usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177

All domains exist and their metadata is readily available using virsh or a
simple Python script. Nova compute service is fully functional, Ceilometer
agent is partially functional as it is able to export compute.node.cpu.*
metrics but nothing related to libvirt domains.

I already filed a bug report
https://bugs.launchpad.net/ceilometer/+bug/1930446, but would appreciate
feedback and/or advice.

Best regards,

Zakhar

---

More deployment information:

Installed Ceilometer-related packages:

ceilometer-agent-compute 2:16.0.0-0ubuntu1~cloud0
ceilometer-common 2:16.0.0-0ubuntu1~cloud0
python3-ceilometer 2:16.0.0-0ubuntu1~cloud0

Installed Nova-related packages:

nova-common 3:23.0.0-0ubuntu1~cloud0
nova-compute 3:23.0.0-0ubuntu1~cloud0
nova-compute-kvm 3:23.0.0-0ubuntu1~cloud0
nova-compute-libvirt 3:23.0.0-0ubuntu1~cloud0
python3-nova 3:23.0.0-0ubuntu1~cloud0
python3-novaclient 2:17.4.0-0ubuntu1~cloud0

Installed Libvirt-related packages:

libvirt-clients 6.0.0-0ubuntu8.9
libvirt-daemon 6.0.0-0ubuntu8.9
libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9
libvirt-daemon-system 6.0.0-0ubuntu8.9
libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9
libvirt0:amd64 6.0.0-0ubuntu8.9
python3-libvirt 6.1.0-1

Installed Qemu-related packages:

libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
qemu-block-extra:amd64 1:4.2-3ubuntu6.16
qemu-kvm 1:4.2-3ubuntu6.16
qemu-system-common 1:4.2-3ubuntu6.16
qemu-system-data 1:4.2-3ubuntu6.16
qemu-system-gui:amd64 1:4.2-3ubuntu6.16
qemu-system-x86 1:4.2-3ubuntu6.16
qemu-utils 1:4.2-3ubuntu6.16

Apparmor is enabled and running the default configuration, no messages
related to apparmor and libvirt, qemu, nova-compute, ceilometer-agent, etc
are visible in the logs. I am also attaching the relevant Ceilometer agent
and Nova configuration files:

ceilometer.conf:

[DEFAULT]
transport_url = rabbit://WORKING-TRANSPORT-URL
verbose = true
debug = true
auth_strategy = keystone
log_dir = /var/log/ceilometer
[compute]
instance_discovery_method = libvirt_metadata
[keystone_authtoken]
www_authenticate_uri = http://CONTROLLER.VIP:5000
auth_url = http://CONTROLLER.VIP:5000
memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer
password = WORKING_PASSWORD
[service_credentials]
auth_type = password
auth_url = http://CONTROLLER.VIP:5000/v3
memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS
project_domain_id = default
user_domain_id = default
project_name = service
username = ceilometer
password = WORKING_PASSWORD
interface = internalURL
region_name = RegionOne
[oslo_messaging_notifications]
driver = messagingv2
transport_url = rabbit://WORKING-TRANSPORT-URL

polling.yaml:

---
sources:
    - name: some_pollsters
      interval: 300
      meters:
        - cpu
        - cpu_l3_cache
        - memory.usage
        - network.incoming.bytes
        - network.incoming.packets
        - network.outgoing.bytes
        - network.outgoing.packets
        - disk.device.read.bytes
        - disk.device.read.requests
        - disk.device.write.bytes
        - disk.device.write.requests
        - hardware.cpu.util
        - hardware.cpu.user
        - hardware.cpu.nice
        - hardware.cpu.system
        - hardware.cpu.idle
        - hardware.cpu.wait
        - hardware.cpu.kernel
        - hardware.cpu.interrupt
        - hardware.memory.used
        - hardware.memory.total
        - hardware.memory.buffer
        - hardware.memory.cached
        - hardware.memory.swap.avail
        - hardware.memory.swap.total
        - hardware.system_stats.io.outgoing.blocks
        - hardware.system_stats.io.incoming.blocks
        - hardware.network.ip.incoming.datagrams
        - hardware.network.ip.outgoing.datagrams

nova.conf:

[DEFAULT]
log_dir = /var/log/nova
lock_path = /var/lock/nova
state_path = /var/lib/nova
instance_usage_audit_period = hour
compute_monitors = cpu.virt_driver,numa_mem_bw.virt_driver
reserved_host_memory_mb = 2048
instance_usage_audit = True
resume_guests_state_on_host_boot = true
my_ip = COMPUTE.HOST.IP.ADDR
report_interval = 30
transport_url = rabbit://WORKING-TRANSPORT-URL
[api]
[api_database]
[barbican]
[cache]
expiration_time = 600
backend = oslo_cache.memcache_pool
backend_argument = memcached_expire_time:660
enabled = true
memcache_servers = LIST-OF-WORKING-MEMCACHED-SERVERS
[cinder]
catalog_info = volumev3::internalURL
[compute]
[conductor]
[console]
[consoleauth]
[cors]
[cyborg]
[database]
connection = mysql+pymysql://WORKING-CONNECTION-STRING
connection_recycle_time = 280
max_pool_size = 5
max_retries = -1
[devices]
[ephemeral_storage_encryption]
[filter_scheduler]
[glance]
api_servers = http://CONTROLLER.VIP:9292
[guestfs]
[healthcheck]
[hyperv]
[image_cache]
[ironic]
[key_manager]
[keystone]
[keystone_authtoken]
www_authenticate_uri = http://CONTROLLER.VIP:5000
auth_url = http://CONTROLLER.VIP:5000
region_name = RegionOne
memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = WORKING-PASSWORD
[libvirt]
live_migration_scheme = ssh
live_migration_permit_post_copy = true
disk_cachemodes="network=writeback,block=writeback"
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = SECRET-UUID
[metrics]
[mks]
[neutron]
auth_url = http://CONTROLLER.VIP:5000
region_name = RegionOne
memcached_servers = LIST-OF-WORKING-MEMCACHED-SERVERS
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = neutron
password = WORKING-PASSWORD
[notifications]
notify_on_state_change = vm_and_task_state
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver = messagingv2
[oslo_messaging_rabbit]
amqp_auto_delete = false
rabbit_ha_queues = true
[oslo_middleware]
[oslo_policy]
[pci]
[placement]
region_name = RegionOne
project_domain_name = Default
project_name = service
auth_type = password
user_domain_name = Default
auth_url = http://CONTROLLER.VIP:5000/v3
username = placement
password = WORKING-PASSWORD
[powervm]
[privsep]
[profiler]
[quota]
[rdp]
[remote_debug]
[scheduler]
[serial_console]
[service_user]
[spice]
[upgrade_levels]
[vault]
[vendordata_dynamic_auth]
[vmware]
[vnc]
enabled = true
server_listen = 0.0.0.0
server_proxyclient_address = $my_ip
novncproxy_base_url = https://WORKING-URL:6080/vnc_auto.html
[workarounds]
[wsgi]
[zvm]
[cells]
enable = False
[os_region_name]
openstack =
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210602/d17c47b8/attachment-0001.html>


More information about the openstack-discuss mailing list