[Openstack] [ceilometer] Troubleshooting Ceilometer

Tom Walsh expresswebsys at gmail.com
Thu Mar 3 18:29:36 UTC 2016


In case anybody runs into this issue in the future, we have determined
what the issue is and how to correct it.

First the ceilometer-agent-notification process seems to be
suppressing the exceptions that were being thrown from the daemon.
This is why the logs didn't seem to shine any light on the issue. The
logs were missing a critical exception message. So I had to run the
agent manually with the --verbose switch:

/usr/bin/python /usr/bin/ceilometer-agent-notification --verbose
--config-file=/etc/ceilometer/ceilometer.conf

Which then yielded this exception:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 214, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py",
line 671, in run_service
    service.start()
  File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 174, in start
    self.event_pipe_manager)
  File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 223, in _configure_main_queue_listeners
    for new_tar in handler.get_targets(cfg.CONF):
  File "/usr/lib/python2.7/dist-packages/ceilometer/ipmi/notifications/ironic.py",
line 87, in get_targets
    for topic in conf.notification_topics]
  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py",
line 2062, in __getattr__
    raise NoSuchOptError(name)
NoSuchOptError: no such option in group DEFAULT: notification_topics

A little quick googling resulted in this patchset:

https://review.openstack.org/#/c/255296/

After applying this patchset to the current ceilometer installation we
are seeing the notifications from the message bus being processed by
the ceilometer agent and then passed to the ceilometer collector.

My guess is that the underlying changes to oslo.messaging have now
broken the config parsing in ceilometer.

It appears that this patchset is missing from the current Ubuntu
cloudarchive ceilometer install (1:5.0.1-0ubuntu1~cloud0)

I hope that helps somebody... I spent enough time on it.

Tom Walsh
ExpressHosting.net




More information about the Openstack mailing list